Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX.] duration: 6.39675778s trace: https://jaeger.dev.storj.io/trace/2e92d97344640fc3 FAIL (2025-11-21 12:01:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 12:00:44.413717038 (291h56m42.965187489s) [upload=257111]: Starting upload Nov 21 12:00:44.413777513 (291h56m42.965247974s) [upload=257111]: Done starting upload Nov 21 12:00:44.413945853 (291h56m42.965416314s) [upload=257111]: Uploading segments... Nov 21 12:00:44.414062085 (291h56m42.965532536s) [upload=257111]: Got next segment. Inline: false Nov 21 12:00:44.414124955 (291h56m42.965595416s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:00:44.414131217 (291h56m42.965601678s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:00:44.659711608 (291h56m43.21118208s) [seg_pos={0 0}, upload=257111]: Begin upload segment... Nov 21 12:00:44.659783275 (291h56m43.211253736s) [seg_pos={0 0}, upload=257111]: Done begin upload segment. Nov 21 12:00:44.659821467 (291h56m43.211291919s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661076968 (291h56m43.212547429s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661623990 (291h56m43.213094451s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662015575 (291h56m43.213486036s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662104145 (291h56m43.213574626s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662331166 (291h56m43.213801617s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662744884 (291h56m43.214215345s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662792254 (291h56m43.214262715s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663124828 (291h56m43.214595289s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663330720 (291h56m43.214801171s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663457621 (291h56m43.214928082s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663871199 (291h56m43.21534165s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663892760 (291h56m43.215363221s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663946742 (291h56m43.215417203s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.664546976 (291h56m43.216017437s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.665139685 (291h56m43.216610136s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:46.269248547 (291h56m44.820718998s) [upload=257111]: Next returned nil segment Nov 21 12:00:46.269292731 (291h56m44.820763192s) [upload=257111]: Waiting for error group managing segments... Nov 21 12:00:46.651938048 (291h56m45.203408509s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.653674856 (291h56m45.205145317s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.656724623 (291h56m45.208195084s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.663448046 (291h56m45.214918497s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.672901036 (291h56m45.224371497s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.677309982 (291h56m45.228780463s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.678373266 (291h56m45.229843727s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.686989964 (291h56m45.238460425s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.690017299 (291h56m45.24148776s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.693246267 (291h56m45.244716728s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.227653148 (291h56m45.779123609s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.255119890 (291h56m45.806590351s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.278989602 (291h56m45.830460064s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.279015171 (291h56m45.830485632s) [seg_pos={0 0}, upload=257111]: Segment reached optimal threshold of 13 pieces. Nov 21 12:00:47.279217887 (291h56m45.830688348s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279226153 (291h56m45.830696614s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279513941 (291h56m45.830984392s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279522787 (291h56m45.830993249s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279754489 (291h56m45.83122495s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:53244->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:00:47.279762644 (291h56m45.831233105s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279782472 (291h56m45.831252934s) [seg_pos={0 0}, upload=257111]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:00:47.279865650 (291h56m45.831336102s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:00:47.279870900 (291h56m45.831341361s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:00:47.570435510 (291h56m46.121905962s) [upload=257111]: Done uploading segments... Nov 21 12:00:47.570460016 (291h56m46.121930478s) [upload=257111]: Upload finished. err: <nil> Duration: 6.396811202s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...X.X......................................................] duration: 5.320575088s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 12:01:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.320611366s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............................] duration: 3.460174567s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 12:01:24 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.460230182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX] duration: 30.003420546s trace: https://jaeger.dev.storj.io/trace/2e92d97344640fc3 FAIL (2025-11-21 12:01:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 12:00:44.413717038 (291h56m42.965187489s) [upload=257111]: Starting upload Nov 21 12:00:44.413777513 (291h56m42.965247974s) [upload=257111]: Done starting upload Nov 21 12:00:44.413945853 (291h56m42.965416314s) [upload=257111]: Uploading segments... Nov 21 12:00:44.414062085 (291h56m42.965532536s) [upload=257111]: Got next segment. Inline: false Nov 21 12:00:44.414124955 (291h56m42.965595416s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:00:44.414131217 (291h56m42.965601678s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:00:44.659711608 (291h56m43.21118208s) [seg_pos={0 0}, upload=257111]: Begin upload segment... Nov 21 12:00:44.659783275 (291h56m43.211253736s) [seg_pos={0 0}, upload=257111]: Done begin upload segment. Nov 21 12:00:44.659821467 (291h56m43.211291919s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661076968 (291h56m43.212547429s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661623990 (291h56m43.213094451s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662015575 (291h56m43.213486036s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662104145 (291h56m43.213574626s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662331166 (291h56m43.213801617s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662744884 (291h56m43.214215345s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662792254 (291h56m43.214262715s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663124828 (291h56m43.214595289s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663330720 (291h56m43.214801171s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663457621 (291h56m43.214928082s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663871199 (291h56m43.21534165s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663892760 (291h56m43.215363221s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663946742 (291h56m43.215417203s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.664546976 (291h56m43.216017437s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.665139685 (291h56m43.216610136s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:46.269248547 (291h56m44.820718998s) [upload=257111]: Next returned nil segment Nov 21 12:00:46.269292731 (291h56m44.820763192s) [upload=257111]: Waiting for error group managing segments... Nov 21 12:00:46.651938048 (291h56m45.203408509s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.653674856 (291h56m45.205145317s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.656724623 (291h56m45.208195084s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.663448046 (291h56m45.214918497s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.672901036 (291h56m45.224371497s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.677309982 (291h56m45.228780463s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.678373266 (291h56m45.229843727s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.686989964 (291h56m45.238460425s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.690017299 (291h56m45.24148776s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.693246267 (291h56m45.244716728s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.227653148 (291h56m45.779123609s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.255119890 (291h56m45.806590351s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.278989602 (291h56m45.830460064s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.279015171 (291h56m45.830485632s) [seg_pos={0 0}, upload=257111]: Segment reached optimal threshold of 13 pieces. Nov 21 12:00:47.279217887 (291h56m45.830688348s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279226153 (291h56m45.830696614s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279513941 (291h56m45.830984392s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279522787 (291h56m45.830993249s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279754489 (291h56m45.83122495s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:53244->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:00:47.279762644 (291h56m45.831233105s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279782472 (291h56m45.831252934s) [seg_pos={0 0}, upload=257111]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:00:47.279865650 (291h56m45.831336102s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:00:47.279870900 (291h56m45.831341361s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:00:47.570435510 (291h56m46.121905962s) [upload=257111]: Done uploading segments... Nov 21 12:00:47.570460016 (291h56m46.121930478s) [upload=257111]: Upload finished. err: <nil> Duration: 30.003448038s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....X.X.....................................................] duration: 6.084197508s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 12:00:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.084233658s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.............................] duration: 4.20643034s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 12:00:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.20646122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.X] duration: 30.006575203s trace: https://jaeger.dev.storj.io/trace/2d0a737d6f0b3a75 FAIL (2025-11-21 12:00:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DEHUL2WNISOUZACYFYT4RLV54ZKCSUYGFXNNKZBCXWGHPWCV4H3Q) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: G5XT2BY35LSKRGSYV3D5Z6R47LVSX5E4TKUWCR4Y6SMXT4W2FQPA) context deadline exceeded ======LOGS====== 4.198056631 (291h55m42.749527092s) [upload=257093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:59:44.444211737 (291h55m42.995682188s) [seg_pos={0 0}, upload=257093]: Begin upload segment... Nov 21 11:59:44.444407079 (291h55m42.99587754s) [seg_pos={0 0}, upload=257093]: Done begin upload segment. Nov 21 11:59:44.444448357 (291h55m42.995918818s) [noise=true , address=77.90.31.12:4589 , piece_id=NQ5CNDUHIVXSO2WIMLFFWB5Q6MV55XJKAUHI3JHX32OA6TJ27LUA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.444723491 (291h55m42.996193952s) [noise=true , address=145.79.227.23:4546 , piece_id=GT7SBNAJWGUUVBOWMJWRFZC6AFT6MESITZOWCBDLQ7UNZ6XUHSZQ, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.445731331 (291h55m42.997201792s) [noise=true , address=77.90.31.19:4842 , piece_id=G5XT2BY35LSKRGSYV3D5Z6R47LVSX5E4TKUWCR4Y6SMXT4W2FQPA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.446336312 (291h55m42.997806773s) [noise=true , address=145.79.227.28:4727 , piece_id=A7JE5X6DB7NAQMTZ3IGOTHZTRCES7PEKIBRDM4E55F25UADXHLGQ, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.446851453 (291h55m42.998321904s) [noise=true , address=145.79.227.20:4437 , piece_id=Y2ZWMWSV42RJP3QDA77XZKQHMZ3VKFC7RSD732JIIHYTTJLW534A, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.447395429 (291h55m42.998865891s) [noise=true , address=77.90.31.13:4628 , piece_id=GPQKPCGQ4XZXS5FFJEQKVGIVD7XUQOFMRHDNOGTGVVLGCQBG4S4A, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.447849113 (291h55m42.999319574s) [noise=true , address=145.79.227.9:4367 , piece_id=WRYDKN4XXSWK3XEPHTECXOGZBXER62TX5W4QB6I3BTM7GAPZOLIA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.448351760 (291h55m42.999822231s) [noise=true , address=145.79.227.26:4653 , piece_id=N2HSKRSHAOMKO3RHAHAT6FRQJD4GGYO5WAJF4HIWG3K22JCPXUZA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.448837004 (291h55m43.000307465s) [noise=true , address=45.91.116.11:4119 , piece_id=DEHUL2WNISOUZACYFYT4RLV54ZKCSUYGFXNNKZBCXWGHPWCV4H3Q, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.450042300 (291h55m43.001512751s) [noise=true , address=145.79.227.22:4510 , piece_id=PTMTSIGJH24IMSYU2LCVNRHGAKVLMKRPVU73OCLZSSJQTAYB4BNA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.450588840 (291h55m43.002059301s) [noise=true , address=145.79.227.8:4329 , piece_id=OAXMUVPXTZFV5YCYU7IYZ4UKQWHNYQ5HYHXIPQMJSE773HKPTNAQ, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.451123318 (291h55m43.002593779s) [noise=true , address=45.91.116.13:4154 , piece_id=CQXYN3CKALT4PIQ2243HJH5YKDGCX5UDGHLFPUIMOMDC2MYYU4YA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.451686801 (291h55m43.003157262s) [noise=true , address=145.79.227.25:4617 , piece_id=5SWWVZFF65GNIUZ7W7EVTDMW6BAYXLBA2FTHAXBUSICSW462WKWA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.452322281 (291h55m43.003792742s) [noise=true , address=145.79.227.13:4801 , piece_id=RMOOF4H6XTER66HQXDHWBFDL5XJESLC6MERUF4N6VQQ3XGVXZAJA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.452883469 (291h55m43.00435393s) [noise=true , address=145.79.227.10:4402 , piece_id=BW73WYCTHSIOV3SB3BIAM6DMLPOQQAC57IRLLV7EHHTDSSHHRL7Q, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.453437975 (291h55m43.004908426s) [noise=true , address=45.91.116.4:4937 , piece_id=HJCGEXX3BKHVUEOX2D4NVYDTQ2ECE2LCJ2T3GHRGF6OUEKFPZMAQ, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:46.149272521 (291h55m44.700742982s) [upload=257093]: Next returned nil segment Nov 21 11:59:46.149287108 (291h55m44.700757569s) [upload=257093]: Waiting for error group managing segments... Nov 21 11:59:46.430745998 (291h55m44.982216459s) [noise=true , address=145.79.227.10:4402 , piece_id=BW73WYCTHSIOV3SB3BIAM6DMLPOQQAC57IRLLV7EHHTDSSHHRL7Q, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.435836702 (291h55m44.987307163s) [noise=true , address=145.79.227.8:4329 , piece_id=OAXMUVPXTZFV5YCYU7IYZ4UKQWHNYQ5HYHXIPQMJSE773HKPTNAQ, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.436122096 (291h55m44.987592557s) [noise=true , address=145.79.227.26:4653 , piece_id=N2HSKRSHAOMKO3RHAHAT6FRQJD4GGYO5WAJF4HIWG3K22JCPXUZA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.437246056 (291h55m44.988716507s) [noise=true , address=145.79.227.25:4617 , piece_id=5SWWVZFF65GNIUZ7W7EVTDMW6BAYXLBA2FTHAXBUSICSW462WKWA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.442802506 (291h55m44.994272977s) [noise=true , address=145.79.227.13:4801 , piece_id=RMOOF4H6XTER66HQXDHWBFDL5XJESLC6MERUF4N6VQQ3XGVXZAJA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.455635032 (291h55m45.007105483s) [noise=true , address=145.79.227.20:4437 , piece_id=Y2ZWMWSV42RJP3QDA77XZKQHMZ3VKFC7RSD732JIIHYTTJLW534A, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.457499862 (291h55m45.008970324s) [noise=true , address=145.79.227.9:4367 , piece_id=WRYDKN4XXSWK3XEPHTECXOGZBXER62TX5W4QB6I3BTM7GAPZOLIA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.459020679 (291h55m45.01049113s) [noise=true , address=145.79.227.22:4510 , piece_id=PTMTSIGJH24IMSYU2LCVNRHGAKVLMKRPVU73OCLZSSJQTAYB4BNA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.459085482 (291h55m45.010555943s) [noise=true , address=145.79.227.28:4727 , piece_id=A7JE5X6DB7NAQMTZ3IGOTHZTRCES7PEKIBRDM4E55F25UADXHLGQ, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.460866684 (291h55m45.012337135s) [noise=true , address=145.79.227.23:4546 , piece_id=GT7SBNAJWGUUVBOWMJWRFZC6AFT6MESITZOWCBDLQ7UNZ6XUHSZQ, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.999746391 (291h55m45.551216843s) [noise=true , address=45.91.116.4:4937 , piece_id=HJCGEXX3BKHVUEOX2D4NVYDTQ2ECE2LCJ2T3GHRGF6OUEKFPZMAQ, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:47.025097955 (291h55m45.576568416s) [noise=true , address=77.90.31.19:4842 , piece_id=G5XT2BY35LSKRGSYV3D5Z6R47LVSX5E4TKUWCR4Y6SMXT4W2FQPA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:47.121865960 (291h55m45.673336431s) [noise=true , address=45.91.116.11:4119 , piece_id=DEHUL2WNISOUZACYFYT4RLV54ZKCSUYGFXNNKZBCXWGHPWCV4H3Q, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:47.121893291 (291h55m45.673363742s) [seg_pos={0 0}, upload=257093]: Segment reached optimal threshold of 13 pieces. Nov 21 11:59:47.122112079 (291h55m45.67358254s) [noise=true , address=77.90.31.13:4628 , piece_id=GPQKPCGQ4XZXS5FFJEQKVGIVD7XUQOFMRHDNOGTGVVLGCQBG4S4A, seg_pos={0 0}, upload=257093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: write tcp 188.166.225.170:55566->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:55566->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55566->77.90.31.13:4628: use of closed network connection Nov 21 11:59:47.122120284 (291h55m45.673590745s) [noise=true , address=77.90.31.13:4628 , piece_id=GPQKPCGQ4XZXS5FFJEQKVGIVD7XUQOFMRHDNOGTGVVLGCQBG4S4A, seg_pos={0 0}, upload=257093]: Long tail cancellation due to optimal threshold Nov 21 11:59:47.122314153 (291h55m45.673784614s) [noise=true , address=45.91.116.13:4154 , piece_id=CQXYN3CKALT4PIQ2243HJH5YKDGCX5UDGHLFPUIMOMDC2MYYU4YA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:46116->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:46116->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46116->45.91.116.13:4154: use of closed network connection Nov 21 11:59:47.122359410 (291h55m45.673829871s) [noise=true , address=45.91.116.13:4154 , piece_id=CQXYN3CKALT4PIQ2243HJH5YKDGCX5UDGHLFPUIMOMDC2MYYU4YA, seg_pos={0 0}, upload=257093]: Long tail cancellation due to optimal threshold Nov 21 11:59:47.122374598 (291h55m45.673845059s) [noise=true , address=77.90.31.12:4589 , piece_id=NQ5CNDUHIVXSO2WIMLFFWB5Q6MV55XJKAUHI3JHX32OA6TJ27LUA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:33090->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:33090->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33090->77.90.31.12:4589: use of closed network connection Nov 21 11:59:47.122386972 (291h55m45.673857423s) [noise=true , address=77.90.31.12:4589 , piece_id=NQ5CNDUHIVXSO2WIMLFFWB5Q6MV55XJKAUHI3JHX32OA6TJ27LUA, seg_pos={0 0}, upload=257093]: Long tail cancellation due to optimal threshold Nov 21 11:59:47.122528912 (291h55m45.673999363s) [seg_pos={0 0}, upload=257093]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:59:47.122877316 (291h55m45.674347777s) [upload=257093]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:59:47.122886784 (291h55m45.674357245s) [upload=257093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:59:47.406105435 (291h55m45.957575896s) [upload=257093]: Done uploading segments... Nov 21 11:59:47.406126023 (291h55m45.957596485s) [upload=257093]: Upload finished. err: <nil> Duration:* 30.006603746s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX............................] duration: 3.39210982s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:59:50 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.392133516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....X.X....................................................] duration: 5.721095522s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:59:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.721129286s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.] duration: 6.403382444s trace: https://jaeger.dev.storj.io/trace/58acb9609da86b42 FAIL (2025-11-21 11:59:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Nov 21 11:58:23.798165294 (291h54m22.349635755s) [upload=257073]: Got next segment. Inline: false Nov 21 11:58:23.798209027 (291h54m22.349679488s) [upload=257073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:58:23.798215149 (291h54m22.34968561s) [upload=257073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:58:24.043750464 (291h54m22.595220925s) [seg_pos={0 0}, upload=257073]: Begin upload segment... Nov 21 11:58:24.043824775 (291h54m22.595295236s) [seg_pos={0 0}, upload=257073]: Done begin upload segment. Nov 21 11:58:24.043886623 (291h54m22.595357084s) [noise=true , address=145.79.227.10:4403 , piece_id=CT6D2DWCPLKM3RMZTHRZ5CGWK5XA4Z3WRFZFQQEARFG5FJRZRG2Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.043914426 (291h54m22.595384887s) [noise=true , address=145.79.227.14:4838 , piece_id=I2R25G47MYOF4ZGAQP6K74XNHPIOHRRJXT56NVTTDRXXJ7TV7H6Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.046406581 (291h54m22.597877052s) [noise=true , address=45.91.116.7:4010 , piece_id=DJIK5ZX2Y65BTPNB72YDZXZREXRNEWVSI2NQR5HHM54OS5QVLM6Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.047221594 (291h54m22.598692045s) [noise=true , address=145.79.227.8:4331 , piece_id=JE53Y3SUDZDUAONS47KOY5R6D6JM3LE4XT7O5HDB34YXKZRS3QZA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.047751973 (291h54m22.599222435s) [noise=true , address=145.79.227.17:4971 , piece_id=6IE5YC2ZZPI23AUMEYNZF5S2WKO77EHR57HRLWK5STZUVAMGNUTQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.048334221 (291h54m22.599804694s) [noise=true , address=145.79.227.23:4546 , piece_id=27GJOAY3EEJ2KQP3OEOC4DQ3N4AE3XK3266RJMQSNEIMCPSAIKIQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.048865253 (291h54m22.600335714s) [noise=true , address=145.79.227.25:4619 , piece_id=5JPN46ZYDROAUDYEYZTEM3W527ZQEDAC5IHOCPYQ2B7J52ZIF2JA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.049423647 (291h54m22.600894108s) [noise=true , address=77.90.31.14:4661 , piece_id=7GNVQHWVEYWHFKFWKOEVPQP73ALNL4CDSOQFTM24MRCKOOK2W72A, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.049965448 (291h54m22.601435909s) [noise=true , address=145.79.227.28:4727 , piece_id=IOWSKVNZYX2W47P2ANIZ42JJ3MZ7HZFKZR6GJAZDZL7REZA2GMBA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.050611228 (291h54m22.602081689s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.051249613 (291h54m22.602720074s) [noise=true , address=145.79.227.15:4875 , piece_id=AR26OLW7DWFIEEVWA2IH3Z7I3QHHK6HEMKLFZ7XVXZXH2MF2RDWQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.051755407 (291h54m22.603225868s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.054542504 (291h54m22.606012965s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.055372274 (291h54m22.606842735s) [noise=true , address=77.90.31.10:4520 , piece_id=UGF46ULG4775AETYWQSVRE4MBV2J5VRBTBYGTVQCZ7C725QWJNPA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.056056407 (291h54m22.607526868s) [noise=true , address=145.79.227.21:4473 , piece_id=FKSBVDDTCH5FQBNGQROSUWGVWKUBDQRDDA7CKQCOYSWHS3LS6WOA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.056876177 (291h54m22.608346638s) [noise=true , address=145.79.227.24:4583 , piece_id=3DSY3NGB522AEBZ2VHPO5DTAVPLYHTN3RVB2MCPJH5OULGSPRNDQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:25.729310655 (291h54m24.280781116s) [upload=257073]: Next returned nil segment Nov 21 11:58:25.729350490 (291h54m24.280820951s) [upload=257073]: Waiting for error group managing segments... Nov 21 11:58:26.033748111 (291h54m24.585218572s) [noise=true , address=145.79.227.10:4403 , piece_id=CT6D2DWCPLKM3RMZTHRZ5CGWK5XA4Z3WRFZFQQEARFG5FJRZRG2Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.040623383 (291h54m24.592093834s) [noise=true , address=145.79.227.17:4971 , piece_id=6IE5YC2ZZPI23AUMEYNZF5S2WKO77EHR57HRLWK5STZUVAMGNUTQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.046295113 (291h54m24.597765564s) [noise=true , address=145.79.227.25:4619 , piece_id=5JPN46ZYDROAUDYEYZTEM3W527ZQEDAC5IHOCPYQ2B7J52ZIF2JA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.050366986 (291h54m24.601837448s) [noise=true , address=145.79.227.14:4838 , piece_id=I2R25G47MYOF4ZGAQP6K74XNHPIOHRRJXT56NVTTDRXXJ7TV7H6Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.051064134 (291h54m24.602534595s) [noise=true , address=145.79.227.8:4331 , piece_id=JE53Y3SUDZDUAONS47KOY5R6D6JM3LE4XT7O5HDB34YXKZRS3QZA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.054474979 (291h54m24.60594544s) [noise=true , address=145.79.227.24:4583 , piece_id=3DSY3NGB522AEBZ2VHPO5DTAVPLYHTN3RVB2MCPJH5OULGSPRNDQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.055010749 (291h54m24.6064812s) [noise=true , address=145.79.227.23:4546 , piece_id=27GJOAY3EEJ2KQP3OEOC4DQ3N4AE3XK3266RJMQSNEIMCPSAIKIQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.056438778 (291h54m24.607909259s) [noise=true , address=145.79.227.21:4473 , piece_id=FKSBVDDTCH5FQBNGQROSUWGVWKUBDQRDDA7CKQCOYSWHS3LS6WOA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.062269401 (291h54m24.613739852s) [noise=true , address=145.79.227.28:4727 , piece_id=IOWSKVNZYX2W47P2ANIZ42JJ3MZ7HZFKZR6GJAZDZL7REZA2GMBA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.073374817 (291h54m24.624845268s) [noise=true , address=145.79.227.15:4875 , piece_id=AR26OLW7DWFIEEVWA2IH3Z7I3QHHK6HEMKLFZ7XVXZXH2MF2RDWQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.559914971 (291h54m25.111385432s) [noise=true , address=77.90.31.10:4520 , piece_id=UGF46ULG4775AETYWQSVRE4MBV2J5VRBTBYGTVQCZ7C725QWJNPA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.590444054 (291h54m25.141914515s) [noise=true , address=77.90.31.14:4661 , piece_id=7GNVQHWVEYWHFKFWKOEVPQP73ALNL4CDSOQFTM24MRCKOOK2W72A, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.592964242 (291h54m25.144434703s) [noise=true , address=45.91.116.7:4010 , piece_id=DJIK5ZX2Y65BTPNB72YDZXZREXRNEWVSI2NQR5HHM54OS5QVLM6Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.592978019 (291h54m25.14444847s) [seg_pos={0 0}, upload=257073]: Segment reached optimal threshold of 13 pieces. Nov 21 11:58:26.593165876 (291h54m25.144636327s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection Nov 21 11:58:26.593172589 (291h54m25.14464305s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593176708 (291h54m25.144647169s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection Nov 21 11:58:26.593188961 (291h54m25.144659422s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593421203 (291h54m25.144891674s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Nov 21 11:58:26.593435059 (291h54m25.14490555s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593459887 (291h54m25.144930348s) [seg_pos={0 0}, upload=257073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:58:26.593570738 (291h54m25.145041189s) [upload=257073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:58:26.593578963 (291h54m25.145049424s) [upload=257073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:58:26.889858055 (291h54m25.441328526s) [upload=257073]: Done uploading segments... Nov 21 11:58:26.889882130 (291h54m25.441352591s) [upload=257073]: Upload finished. err: <nil> Duration: 6.403403694s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [......X.X...................................................] duration: 8.92992406s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:59:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 8.929944509s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X] duration: 30.003520926s trace: https://jaeger.dev.storj.io/trace/58acb9609da86b42 FAIL (2025-11-21 11:58:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Nov 21 11:58:23.798165294 (291h54m22.349635755s) [upload=257073]: Got next segment. Inline: false Nov 21 11:58:23.798209027 (291h54m22.349679488s) [upload=257073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:58:23.798215149 (291h54m22.34968561s) [upload=257073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:58:24.043750464 (291h54m22.595220925s) [seg_pos={0 0}, upload=257073]: Begin upload segment... Nov 21 11:58:24.043824775 (291h54m22.595295236s) [seg_pos={0 0}, upload=257073]: Done begin upload segment. Nov 21 11:58:24.043886623 (291h54m22.595357084s) [noise=true , address=145.79.227.10:4403 , piece_id=CT6D2DWCPLKM3RMZTHRZ5CGWK5XA4Z3WRFZFQQEARFG5FJRZRG2Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.043914426 (291h54m22.595384887s) [noise=true , address=145.79.227.14:4838 , piece_id=I2R25G47MYOF4ZGAQP6K74XNHPIOHRRJXT56NVTTDRXXJ7TV7H6Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.046406581 (291h54m22.597877052s) [noise=true , address=45.91.116.7:4010 , piece_id=DJIK5ZX2Y65BTPNB72YDZXZREXRNEWVSI2NQR5HHM54OS5QVLM6Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.047221594 (291h54m22.598692045s) [noise=true , address=145.79.227.8:4331 , piece_id=JE53Y3SUDZDUAONS47KOY5R6D6JM3LE4XT7O5HDB34YXKZRS3QZA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.047751973 (291h54m22.599222435s) [noise=true , address=145.79.227.17:4971 , piece_id=6IE5YC2ZZPI23AUMEYNZF5S2WKO77EHR57HRLWK5STZUVAMGNUTQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.048334221 (291h54m22.599804694s) [noise=true , address=145.79.227.23:4546 , piece_id=27GJOAY3EEJ2KQP3OEOC4DQ3N4AE3XK3266RJMQSNEIMCPSAIKIQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.048865253 (291h54m22.600335714s) [noise=true , address=145.79.227.25:4619 , piece_id=5JPN46ZYDROAUDYEYZTEM3W527ZQEDAC5IHOCPYQ2B7J52ZIF2JA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.049423647 (291h54m22.600894108s) [noise=true , address=77.90.31.14:4661 , piece_id=7GNVQHWVEYWHFKFWKOEVPQP73ALNL4CDSOQFTM24MRCKOOK2W72A, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.049965448 (291h54m22.601435909s) [noise=true , address=145.79.227.28:4727 , piece_id=IOWSKVNZYX2W47P2ANIZ42JJ3MZ7HZFKZR6GJAZDZL7REZA2GMBA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.050611228 (291h54m22.602081689s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.051249613 (291h54m22.602720074s) [noise=true , address=145.79.227.15:4875 , piece_id=AR26OLW7DWFIEEVWA2IH3Z7I3QHHK6HEMKLFZ7XVXZXH2MF2RDWQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.051755407 (291h54m22.603225868s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.054542504 (291h54m22.606012965s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.055372274 (291h54m22.606842735s) [noise=true , address=77.90.31.10:4520 , piece_id=UGF46ULG4775AETYWQSVRE4MBV2J5VRBTBYGTVQCZ7C725QWJNPA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.056056407 (291h54m22.607526868s) [noise=true , address=145.79.227.21:4473 , piece_id=FKSBVDDTCH5FQBNGQROSUWGVWKUBDQRDDA7CKQCOYSWHS3LS6WOA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.056876177 (291h54m22.608346638s) [noise=true , address=145.79.227.24:4583 , piece_id=3DSY3NGB522AEBZ2VHPO5DTAVPLYHTN3RVB2MCPJH5OULGSPRNDQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:25.729310655 (291h54m24.280781116s) [upload=257073]: Next returned nil segment Nov 21 11:58:25.729350490 (291h54m24.280820951s) [upload=257073]: Waiting for error group managing segments... Nov 21 11:58:26.033748111 (291h54m24.585218572s) [noise=true , address=145.79.227.10:4403 , piece_id=CT6D2DWCPLKM3RMZTHRZ5CGWK5XA4Z3WRFZFQQEARFG5FJRZRG2Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.040623383 (291h54m24.592093834s) [noise=true , address=145.79.227.17:4971 , piece_id=6IE5YC2ZZPI23AUMEYNZF5S2WKO77EHR57HRLWK5STZUVAMGNUTQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.046295113 (291h54m24.597765564s) [noise=true , address=145.79.227.25:4619 , piece_id=5JPN46ZYDROAUDYEYZTEM3W527ZQEDAC5IHOCPYQ2B7J52ZIF2JA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.050366986 (291h54m24.601837448s) [noise=true , address=145.79.227.14:4838 , piece_id=I2R25G47MYOF4ZGAQP6K74XNHPIOHRRJXT56NVTTDRXXJ7TV7H6Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.051064134 (291h54m24.602534595s) [noise=true , address=145.79.227.8:4331 , piece_id=JE53Y3SUDZDUAONS47KOY5R6D6JM3LE4XT7O5HDB34YXKZRS3QZA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.054474979 (291h54m24.60594544s) [noise=true , address=145.79.227.24:4583 , piece_id=3DSY3NGB522AEBZ2VHPO5DTAVPLYHTN3RVB2MCPJH5OULGSPRNDQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.055010749 (291h54m24.6064812s) [noise=true , address=145.79.227.23:4546 , piece_id=27GJOAY3EEJ2KQP3OEOC4DQ3N4AE3XK3266RJMQSNEIMCPSAIKIQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.056438778 (291h54m24.607909259s) [noise=true , address=145.79.227.21:4473 , piece_id=FKSBVDDTCH5FQBNGQROSUWGVWKUBDQRDDA7CKQCOYSWHS3LS6WOA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.062269401 (291h54m24.613739852s) [noise=true , address=145.79.227.28:4727 , piece_id=IOWSKVNZYX2W47P2ANIZ42JJ3MZ7HZFKZR6GJAZDZL7REZA2GMBA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.073374817 (291h54m24.624845268s) [noise=true , address=145.79.227.15:4875 , piece_id=AR26OLW7DWFIEEVWA2IH3Z7I3QHHK6HEMKLFZ7XVXZXH2MF2RDWQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.559914971 (291h54m25.111385432s) [noise=true , address=77.90.31.10:4520 , piece_id=UGF46ULG4775AETYWQSVRE4MBV2J5VRBTBYGTVQCZ7C725QWJNPA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.590444054 (291h54m25.141914515s) [noise=true , address=77.90.31.14:4661 , piece_id=7GNVQHWVEYWHFKFWKOEVPQP73ALNL4CDSOQFTM24MRCKOOK2W72A, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.592964242 (291h54m25.144434703s) [noise=true , address=45.91.116.7:4010 , piece_id=DJIK5ZX2Y65BTPNB72YDZXZREXRNEWVSI2NQR5HHM54OS5QVLM6Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.592978019 (291h54m25.14444847s) [seg_pos={0 0}, upload=257073]: Segment reached optimal threshold of 13 pieces. Nov 21 11:58:26.593165876 (291h54m25.144636327s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection Nov 21 11:58:26.593172589 (291h54m25.14464305s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593176708 (291h54m25.144647169s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection Nov 21 11:58:26.593188961 (291h54m25.144659422s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593421203 (291h54m25.144891674s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Nov 21 11:58:26.593435059 (291h54m25.14490555s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593459887 (291h54m25.144930348s) [seg_pos={0 0}, upload=257073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:58:26.593570738 (291h54m25.145041189s) [upload=257073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:58:26.593578963 (291h54m25.145049424s) [upload=257073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:58:26.889858055 (291h54m25.441328526s) [upload=257073]: Done uploading segments... Nov 21 11:58:26.889882130 (291h54m25.441352591s) [upload=257073]: Upload finished. err: <nil> Duration: 30.003541314s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.......X.X..................................................] duration: 5.74198459s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:58:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.74201595s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........................] duration: 3.705145897s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:58:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.705175944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........X.X.................................................] duration: 6.74982695s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:58:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.749855766s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........................] duration: 3.583500704s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:58:09 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.583528697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.........X.X................................................] duration: 5.755210397s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:57:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.755239783s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.] duration: 6.277714495s trace: https://jaeger.dev.storj.io/trace/7fdfc4a3d7b030f FAIL (2025-11-21 11:57:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 11:56:49.062426520 (291h52m47.613896981s) [upload=257044]: Starting upload Nov 21 11:56:49.062524096 (291h52m47.613994557s) [upload=257044]: Done starting upload Nov 21 11:56:49.062647982 (291h52m47.614118444s) [upload=257044]: Uploading segments... Nov 21 11:56:49.062821623 (291h52m47.614292074s) [upload=257044]: Got next segment. Inline: false Nov 21 11:56:49.062899020 (291h52m47.614369501s) [upload=257044]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:56:49.062905733 (291h52m47.614376204s) [upload=257044]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:56:49.309543198 (291h52m47.861013659s) [seg_pos={0 0}, upload=257044]: Begin upload segment... Nov 21 11:56:49.309603794 (291h52m47.861074245s) [seg_pos={0 0}, upload=257044]: Done begin upload segment. Nov 21 11:56:49.309643459 (291h52m47.86111391s) [noise=true , address=45.91.116.11:4117 , piece_id=YIA5ZDJHFNHJ56HF3D7POCP5LS2QRKO2V3BNZ2PEVOSG5KKXNUAA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.309770822 (291h52m47.861241283s) [noise=true , address=145.79.227.11:4729 , piece_id=YE2ULEWMKHW5OWOFVZ3TUGIVOILKMLHCOKFW34NBNTX5VVO7NFBA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.310671858 (291h52m47.862142319s) [noise=true , address=77.90.31.18:4807 , piece_id=VC5S4SVBMAE7PYLDZPXNI7GEYXZHV2F5KXNOSK4DKSBH4LYCIN5Q, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.311167391 (291h52m47.862637842s) [noise=true , address=77.90.31.14:4661 , piece_id=HPXKLWYM7VQJ5JELZPCY2GEIC5M7ZAAR6RN4GCUYUMN54IWRZ2GA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.311643197 (291h52m47.863113659s) [noise=true , address=145.79.227.25:4617 , piece_id=NVC7CYKQN5JUDANSLU3OSTDYMD4ZDJTLLWHVS47RWNZCY6Z4XZ2Q, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.312162225 (291h52m47.863632676s) [noise=true , address=145.79.227.23:4547 , piece_id=E3SO2EJ33JKCXO23ZWKLSUITQPASWANW3FFSC7AN4RX3ZYFT2SZA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.312704428 (291h52m47.864174889s) [noise=true , address=145.79.227.16:4909 , piece_id=KGTBQN3SHG2JBQ3CXHX7JPURPCEL3U7Z2NRN2ZWH5ZTZRFOH7DRA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.313220360 (291h52m47.864690811s) [noise=true , address=45.91.116.5:4298 , piece_id=ZS53VM2B462JNIRZYJEZR2AUOOB3WMXBC5YUFZR3GWTV4FVHM2YA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.314776784 (291h52m47.866247245s) [noise=true , address=145.79.227.30:4761 , piece_id=E3DWGFRIEMHCDWQOJ66IE5O6QU7TE3A3AAVWF6BSP472KPP6VAFQ, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.314974149 (291h52m47.866444611s) [noise=true , address=145.79.227.20:4439 , piece_id=TNBZK4G7DTDB4OWRKGEBVTQG4WUXYPAKVB3YFWAPSC22DZPDFZKQ, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.316552255 (291h52m47.868022706s) [noise=true , address=145.79.227.30:4763 , piece_id=XD67LA6SWX5LSTXURPLBHFVTELZ63OMKNE7YYGWP764NTVVAMZ2Q, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.318437675 (291h52m47.869908136s) [noise=true , address=45.91.116.4:4940 , piece_id=ASYFJLVD5Q2MYT3CBKXNRF4DC2WW7WQPFTHOUAY3RWTHTNKSP5MQ, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.320554105 (291h52m47.872024576s) [noise=true , address=77.90.31.10:4517 , piece_id=N57SZLF5LKFHXCMS7AIKO7SQRNR7SQGPA73CRIBKN6YTGHPTUI4Q, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.323834612 (291h52m47.875305073s) [noise=true , address=77.90.31.12:4592 , piece_id=DDAIQZ4QVXGNXHX7BKXQSJ2IWPL5KH2LN3IYPFL2LMMG2KPAJSQA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.326521458 (291h52m47.877991919s) [noise=true , address=145.79.227.26:4653 , piece_id=HFJIOCZ72DN7YALL7GGVILZY6OMMR5D643GJFTIHCGZKQB2D7FAA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.328754119 (291h52m47.88022459s) [noise=true , address=145.79.227.15:4876 , piece_id=FUFMDOYCT7MJ2EMKRD5FJFOBW24QB6C3FJT7OIGVOKJZ5BEC24ZA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:51.023648269 (291h52m49.57511873s) [upload=257044]: Next returned nil segment Nov 21 11:56:51.023664790 (291h52m49.575135251s) [upload=257044]: Waiting for error group managing segments... Nov 21 11:56:51.309154256 (291h52m49.860624717s) [noise=true , address=145.79.227.20:4439 , piece_id=TNBZK4G7DTDB4OWRKGEBVTQG4WUXYPAKVB3YFWAPSC22DZPDFZKQ, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.310683438 (291h52m49.862153899s) [noise=true , address=145.79.227.16:4909 , piece_id=KGTBQN3SHG2JBQ3CXHX7JPURPCEL3U7Z2NRN2ZWH5ZTZRFOH7DRA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.310803197 (291h52m49.862273658s) [noise=true , address=145.79.227.25:4617 , piece_id=NVC7CYKQN5JUDANSLU3OSTDYMD4ZDJTLLWHVS47RWNZCY6Z4XZ2Q, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.320217753 (291h52m49.871688214s) [noise=true , address=145.79.227.23:4547 , piece_id=E3SO2EJ33JKCXO23ZWKLSUITQPASWANW3FFSC7AN4RX3ZYFT2SZA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.324821500 (291h52m49.876291951s) [noise=true , address=145.79.227.26:4653 , piece_id=HFJIOCZ72DN7YALL7GGVILZY6OMMR5D643GJFTIHCGZKQB2D7FAA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.328048665 (291h52m49.879519126s) [noise=true , address=145.79.227.30:4761 , piece_id=E3DWGFRIEMHCDWQOJ66IE5O6QU7TE3A3AAVWF6BSP472KPP6VAFQ, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.330439999 (291h52m49.88191046s) [noise=true , address=145.79.227.30:4763 , piece_id=XD67LA6SWX5LSTXURPLBHFVTELZ63OMKNE7YYGWP764NTVVAMZ2Q, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.339147259 (291h52m49.89061772s) [noise=true , address=145.79.227.15:4876 , piece_id=FUFMDOYCT7MJ2EMKRD5FJFOBW24QB6C3FJT7OIGVOKJZ5BEC24ZA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.352036012 (291h52m49.903506473s) [noise=true , address=145.79.227.11:4729 , piece_id=YE2ULEWMKHW5OWOFVZ3TUGIVOILKMLHCOKFW34NBNTX5VVO7NFBA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:52.834428138 (291h52m51.385898599s) [noise=true , address=77.90.31.10:4517 , piece_id=N57SZLF5LKFHXCMS7AIKO7SQRNR7SQGPA73CRIBKN6YTGHPTUI4Q, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:52.897703496 (291h52m51.449173957s) [noise=true , address=77.90.31.12:4592 , piece_id=DDAIQZ4QVXGNXHX7BKXQSJ2IWPL5KH2LN3IYPFL2LMMG2KPAJSQA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:52.900050415 (291h52m51.451520876s) [noise=true , address=45.91.116.4:4940 , piece_id=ASYFJLVD5Q2MYT3CBKXNRF4DC2WW7WQPFTHOUAY3RWTHTNKSP5MQ, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:52.900671478 (291h52m51.452141939s) [noise=true , address=45.91.116.5:4298 , piece_id=ZS53VM2B462JNIRZYJEZR2AUOOB3WMXBC5YUFZR3GWTV4FVHM2YA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:52.900683330 (291h52m51.452153791s) [seg_pos={0 0}, upload=257044]: Segment reached optimal threshold of 13 pieces. Nov 21 11:56:52.901024770 (291h52m51.452495241s) [noise=true , address=45.91.116.11:4117 , piece_id=YIA5ZDJHFNHJ56HF3D7POCP5LS2QRKO2V3BNZ2PEVOSG5KKXNUAA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: read tcp 188.166.225.170:41022->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 11:56:52.901034920 (291h52m51.452505381s) [noise=true , address=45.91.116.11:4117 , piece_id=YIA5ZDJHFNHJ56HF3D7POCP5LS2QRKO2V3BNZ2PEVOSG5KKXNUAA, seg_pos={0 0}, upload=257044]: Long tail cancellation due to optimal threshold Nov 21 11:56:52.901387612 (291h52m51.452858073s) [noise=true , address=77.90.31.14:4661 , piece_id=HPXKLWYM7VQJ5JELZPCY2GEIC5M7ZAAR6RN4GCUYUMN54IWRZ2GA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Nov 21 11:56:52.901397170 (291h52m51.452867631s) [noise=true , address=77.90.31.14:4661 , piece_id=HPXKLWYM7VQJ5JELZPCY2GEIC5M7ZAAR6RN4GCUYUMN54IWRZ2GA, seg_pos={0 0}, upload=257044]: Long tail cancellation due to optimal threshold Nov 21 11:56:52.901645422 (291h52m51.453115893s) [noise=true , address=77.90.31.18:4807 , piece_id=VC5S4SVBMAE7PYLDZPXNI7GEYXZHV2F5KXNOSK4DKSBH4LYCIN5Q, seg_pos={0 0}, upload=257044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: read tcp 188.166.225.170:44540->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 11:56:52.901655421 (291h52m51.453125882s) [noise=true , address=77.90.31.18:4807 , piece_id=VC5S4SVBMAE7PYLDZPXNI7GEYXZHV2F5KXNOSK4DKSBH4LYCIN5Q, seg_pos={0 0}, upload=257044]: Long tail cancellation due to optimal threshold Nov 21 11:56:52.901675520 (291h52m51.453145981s) [seg_pos={0 0}, upload=257044]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:56:52.901761854 (291h52m51.453232325s) [upload=257044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:56:52.901768206 (291h52m51.453238668s) [upload=257044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:56:53.209099544 (291h52m51.760569995s) [upload=257044]: Done uploading segments... Nov 21 11:56:53.209124922 (291h52m51.760595383s) [upload=257044]: Upload finished. err: <nil> Duration: 6.277745163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X.XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX] duration: 30.002757091s trace: https://jaeger.dev.storj.io/trace/7fdfc4a3d7b030f FAIL (2025-11-21 11:57:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 11:56:49.062426520 (291h52m47.613896981s) [upload=257044]: Starting upload Nov 21 11:56:49.062524096 (291h52m47.613994557s) [upload=257044]: Done starting upload Nov 21 11:56:49.062647982 (291h52m47.614118444s) [upload=257044]: Uploading segments... Nov 21 11:56:49.062821623 (291h52m47.614292074s) [upload=257044]: Got next segment. Inline: false Nov 21 11:56:49.062899020 (291h52m47.614369501s) [upload=257044]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:56:49.062905733 (291h52m47.614376204s) [upload=257044]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:56:49.309543198 (291h52m47.861013659s) [seg_pos={0 0}, upload=257044]: Begin upload segment... Nov 21 11:56:49.309603794 (291h52m47.861074245s) [seg_pos={0 0}, upload=257044]: Done begin upload segment. Nov 21 11:56:49.309643459 (291h52m47.86111391s) [noise=true , address=45.91.116.11:4117 , piece_id=YIA5ZDJHFNHJ56HF3D7POCP5LS2QRKO2V3BNZ2PEVOSG5KKXNUAA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.309770822 (291h52m47.861241283s) [noise=true , address=145.79.227.11:4729 , piece_id=YE2ULEWMKHW5OWOFVZ3TUGIVOILKMLHCOKFW34NBNTX5VVO7NFBA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.310671858 (291h52m47.862142319s) [noise=true , address=77.90.31.18:4807 , piece_id=VC5S4SVBMAE7PYLDZPXNI7GEYXZHV2F5KXNOSK4DKSBH4LYCIN5Q, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.311167391 (291h52m47.862637842s) [noise=true , address=77.90.31.14:4661 , piece_id=HPXKLWYM7VQJ5JELZPCY2GEIC5M7ZAAR6RN4GCUYUMN54IWRZ2GA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.311643197 (291h52m47.863113659s) [noise=true , address=145.79.227.25:4617 , piece_id=NVC7CYKQN5JUDANSLU3OSTDYMD4ZDJTLLWHVS47RWNZCY6Z4XZ2Q, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.312162225 (291h52m47.863632676s) [noise=true , address=145.79.227.23:4547 , piece_id=E3SO2EJ33JKCXO23ZWKLSUITQPASWANW3FFSC7AN4RX3ZYFT2SZA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.312704428 (291h52m47.864174889s) [noise=true , address=145.79.227.16:4909 , piece_id=KGTBQN3SHG2JBQ3CXHX7JPURPCEL3U7Z2NRN2ZWH5ZTZRFOH7DRA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.313220360 (291h52m47.864690811s) [noise=true , address=45.91.116.5:4298 , piece_id=ZS53VM2B462JNIRZYJEZR2AUOOB3WMXBC5YUFZR3GWTV4FVHM2YA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.314776784 (291h52m47.866247245s) [noise=true , address=145.79.227.30:4761 , piece_id=E3DWGFRIEMHCDWQOJ66IE5O6QU7TE3A3AAVWF6BSP472KPP6VAFQ, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.314974149 (291h52m47.866444611s) [noise=true , address=145.79.227.20:4439 , piece_id=TNBZK4G7DTDB4OWRKGEBVTQG4WUXYPAKVB3YFWAPSC22DZPDFZKQ, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.316552255 (291h52m47.868022706s) [noise=true , address=145.79.227.30:4763 , piece_id=XD67LA6SWX5LSTXURPLBHFVTELZ63OMKNE7YYGWP764NTVVAMZ2Q, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.318437675 (291h52m47.869908136s) [noise=true , address=45.91.116.4:4940 , piece_id=ASYFJLVD5Q2MYT3CBKXNRF4DC2WW7WQPFTHOUAY3RWTHTNKSP5MQ, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.320554105 (291h52m47.872024576s) [noise=true , address=77.90.31.10:4517 , piece_id=N57SZLF5LKFHXCMS7AIKO7SQRNR7SQGPA73CRIBKN6YTGHPTUI4Q, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.323834612 (291h52m47.875305073s) [noise=true , address=77.90.31.12:4592 , piece_id=DDAIQZ4QVXGNXHX7BKXQSJ2IWPL5KH2LN3IYPFL2LMMG2KPAJSQA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.326521458 (291h52m47.877991919s) [noise=true , address=145.79.227.26:4653 , piece_id=HFJIOCZ72DN7YALL7GGVILZY6OMMR5D643GJFTIHCGZKQB2D7FAA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:49.328754119 (291h52m47.88022459s) [noise=true , address=145.79.227.15:4876 , piece_id=FUFMDOYCT7MJ2EMKRD5FJFOBW24QB6C3FJT7OIGVOKJZ5BEC24ZA, seg_pos={0 0}, upload=257044]: Uploading piece... Nov 21 11:56:51.023648269 (291h52m49.57511873s) [upload=257044]: Next returned nil segment Nov 21 11:56:51.023664790 (291h52m49.575135251s) [upload=257044]: Waiting for error group managing segments... Nov 21 11:56:51.309154256 (291h52m49.860624717s) [noise=true , address=145.79.227.20:4439 , piece_id=TNBZK4G7DTDB4OWRKGEBVTQG4WUXYPAKVB3YFWAPSC22DZPDFZKQ, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.310683438 (291h52m49.862153899s) [noise=true , address=145.79.227.16:4909 , piece_id=KGTBQN3SHG2JBQ3CXHX7JPURPCEL3U7Z2NRN2ZWH5ZTZRFOH7DRA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.310803197 (291h52m49.862273658s) [noise=true , address=145.79.227.25:4617 , piece_id=NVC7CYKQN5JUDANSLU3OSTDYMD4ZDJTLLWHVS47RWNZCY6Z4XZ2Q, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.320217753 (291h52m49.871688214s) [noise=true , address=145.79.227.23:4547 , piece_id=E3SO2EJ33JKCXO23ZWKLSUITQPASWANW3FFSC7AN4RX3ZYFT2SZA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.324821500 (291h52m49.876291951s) [noise=true , address=145.79.227.26:4653 , piece_id=HFJIOCZ72DN7YALL7GGVILZY6OMMR5D643GJFTIHCGZKQB2D7FAA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.328048665 (291h52m49.879519126s) [noise=true , address=145.79.227.30:4761 , piece_id=E3DWGFRIEMHCDWQOJ66IE5O6QU7TE3A3AAVWF6BSP472KPP6VAFQ, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.330439999 (291h52m49.88191046s) [noise=true , address=145.79.227.30:4763 , piece_id=XD67LA6SWX5LSTXURPLBHFVTELZ63OMKNE7YYGWP764NTVVAMZ2Q, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.339147259 (291h52m49.89061772s) [noise=true , address=145.79.227.15:4876 , piece_id=FUFMDOYCT7MJ2EMKRD5FJFOBW24QB6C3FJT7OIGVOKJZ5BEC24ZA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:51.352036012 (291h52m49.903506473s) [noise=true , address=145.79.227.11:4729 , piece_id=YE2ULEWMKHW5OWOFVZ3TUGIVOILKMLHCOKFW34NBNTX5VVO7NFBA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:52.834428138 (291h52m51.385898599s) [noise=true , address=77.90.31.10:4517 , piece_id=N57SZLF5LKFHXCMS7AIKO7SQRNR7SQGPA73CRIBKN6YTGHPTUI4Q, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:52.897703496 (291h52m51.449173957s) [noise=true , address=77.90.31.12:4592 , piece_id=DDAIQZ4QVXGNXHX7BKXQSJ2IWPL5KH2LN3IYPFL2LMMG2KPAJSQA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:52.900050415 (291h52m51.451520876s) [noise=true , address=45.91.116.4:4940 , piece_id=ASYFJLVD5Q2MYT3CBKXNRF4DC2WW7WQPFTHOUAY3RWTHTNKSP5MQ, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:52.900671478 (291h52m51.452141939s) [noise=true , address=45.91.116.5:4298 , piece_id=ZS53VM2B462JNIRZYJEZR2AUOOB3WMXBC5YUFZR3GWTV4FVHM2YA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: <nil> Nov 21 11:56:52.900683330 (291h52m51.452153791s) [seg_pos={0 0}, upload=257044]: Segment reached optimal threshold of 13 pieces. Nov 21 11:56:52.901024770 (291h52m51.452495241s) [noise=true , address=45.91.116.11:4117 , piece_id=YIA5ZDJHFNHJ56HF3D7POCP5LS2QRKO2V3BNZ2PEVOSG5KKXNUAA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: read tcp 188.166.225.170:41022->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 11:56:52.901034920 (291h52m51.452505381s) [noise=true , address=45.91.116.11:4117 , piece_id=YIA5ZDJHFNHJ56HF3D7POCP5LS2QRKO2V3BNZ2PEVOSG5KKXNUAA, seg_pos={0 0}, upload=257044]: Long tail cancellation due to optimal threshold Nov 21 11:56:52.901387612 (291h52m51.452858073s) [noise=true , address=77.90.31.14:4661 , piece_id=HPXKLWYM7VQJ5JELZPCY2GEIC5M7ZAAR6RN4GCUYUMN54IWRZ2GA, seg_pos={0 0}, upload=257044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Nov 21 11:56:52.901397170 (291h52m51.452867631s) [noise=true , address=77.90.31.14:4661 , piece_id=HPXKLWYM7VQJ5JELZPCY2GEIC5M7ZAAR6RN4GCUYUMN54IWRZ2GA, seg_pos={0 0}, upload=257044]: Long tail cancellation due to optimal threshold Nov 21 11:56:52.901645422 (291h52m51.453115893s) [noise=true , address=77.90.31.18:4807 , piece_id=VC5S4SVBMAE7PYLDZPXNI7GEYXZHV2F5KXNOSK4DKSBH4LYCIN5Q, seg_pos={0 0}, upload=257044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: read tcp 188.166.225.170:44540->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 11:56:52.901655421 (291h52m51.453125882s) [noise=true , address=77.90.31.18:4807 , piece_id=VC5S4SVBMAE7PYLDZPXNI7GEYXZHV2F5KXNOSK4DKSBH4LYCIN5Q, seg_pos={0 0}, upload=257044]: Long tail cancellation due to optimal threshold Nov 21 11:56:52.901675520 (291h52m51.453145981s) [seg_pos={0 0}, upload=257044]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:56:52.901761854 (291h52m51.453232325s) [upload=257044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:56:52.901768206 (291h52m51.453238668s) [upload=257044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:56:53.209099544 (291h52m51.760569995s) [upload=257044]: Done uploading segments... Nov 21 11:56:53.209124922 (291h52m51.760595383s) [upload=257044]: Upload finished. err: <nil> Duration: 30.002781878s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..........X.X...............................................] duration: 5.23536588s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:57:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.235394093s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........................] duration: 3.674666923s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:56:58 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.674695628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...........X.X..............................................] duration: 8.24736139s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:56:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 8.247392297s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X.XX...X.X.X....X............X....X.X.X..X.X..X...X..XX] duration: 30.003876259s trace: https://jaeger.dev.storj.io/trace/2a9dd7da8f4598d5 FAIL (2025-11-21 11:56:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: SFE5I7QNXOTRI6RUOFRSH3ROOGVT2XQYDKTXXM5AX7X7BVNCX7KQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: X7GQX3EDUFSPKGMWWI755MITJZEWIZSETYNII7WY6HZPSEVHSKMQ) context deadline exceeded ======LOGS====== Nov 21 11:55:43.552695696 (291h51m42.104166147s) [upload=257030]: Starting upload Nov 21 11:55:43.552810114 (291h51m42.104280575s) [upload=257030]: Done starting upload Nov 21 11:55:43.552919241 (291h51m42.104389723s) [upload=257030]: Uploading segments... Nov 21 11:55:43.553102771 (291h51m42.104573242s) [upload=257030]: Got next segment. Inline: false Nov 21 11:55:43.553148118 (291h51m42.104618589s) [upload=257030]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:55:43.553154019 (291h51m42.10462448s) [upload=257030]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:55:43.802668351 (291h51m42.354138812s) [seg_pos={0 0}, upload=257030]: Begin upload segment... Nov 21 11:55:43.802731552 (291h51m42.354202023s) [seg_pos={0 0}, upload=257030]: Done begin upload segment. Nov 21 11:55:43.802743845 (291h51m42.354214296s) [noise=true , address=45.91.116.13:4154 , piece_id=SFE5I7QNXOTRI6RUOFRSH3ROOGVT2XQYDKTXXM5AX7X7BVNCX7KQ, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.802832675 (291h51m42.354303136s) [noise=true , address=145.79.227.23:4547 , piece_id=ZUBVHF47DA7YN4KMRW6JAEGKQ2B3ERPXCRKTYGGO3NBPI62VBAZQ, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.803351683 (291h51m42.354822134s) [noise=true , address=145.79.227.31:4797 , piece_id=IM7URIOGHOOBAQJH737TVDHC65ME35CVY2B7CQMTWNDML6J4TU2Q, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.803641314 (291h51m42.355111765s) [noise=true , address=145.79.227.17:4970 , piece_id=YTH4UCJPW4QY5L3KAIMGRUCRLWQDIVD3EUH5KR6UI6HJQ57VKUCA, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.804028060 (291h51m42.355498522s) [noise=true , address=145.79.227.21:4475 , piece_id=4KJ6F2VL2XYSM4LAVSLGI5G4YD47CPYCCF5D6VO6737T74WUHVVQ, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.804719627 (291h51m42.356190099s) [noise=true , address=77.90.31.17:4771 , piece_id=DEJCDBH4M4MXKYML5JNHHULV7M77VYROL5HYAZZKESLQY6USBL3Q, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.805177279 (291h51m42.35664774s) [noise=true , address=145.79.227.16:4911 , piece_id=U7C7MJOGXDCKAGRLPHVHT4BLYD7X3QPJVHZAADXYA2P4PN2D56YA, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.805723078 (291h51m42.357193539s) [noise=true , address=145.79.227.10:4401 , piece_id=SSMPPQJHJJREY43ZY6HHZTNPJ2AW4M2TTKQTTXJY72EZ53D4YHYQ, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.807345237 (291h51m42.358815688s) [noise=true , address=145.79.227.11:4731 , piece_id=MHJZ3XCIOQSIBUGGBESKSB4YUNL2MRWF7KAB3DNOJXY4YAJUU4DQ, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.808485719 (291h51m42.35995618s) [noise=true , address=145.79.227.27:4689 , piece_id=K64ZSMBDDY2YLEZNQR7R43CTNSEPV6EBFLOLHCUPWXKAU63EUBFA, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.812729520 (291h51m42.364199981s) [noise=true , address=77.90.31.13:4626 , piece_id=X7GQX3EDUFSPKGMWWI755MITJZEWIZSETYNII7WY6HZPSEVHSKMQ, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.813479588 (291h51m42.364950049s) [noise=true , address=145.79.227.22:4509 , piece_id=JU7XTRZGA6UTOWQ2T7ELDVQ43LPVFZGJS4CHZHO5UKZQZOIHDL2A, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.816051806 (291h51m42.367522257s) [noise=true , address=145.79.227.23:4546 , piece_id=RHP7FMWGYLV3DWZ67JSYRR4K74TCGZ756VRCCYWUX7OVZ57URZFA, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.818666866 (291h51m42.370137327s) [noise=true , address=145.79.227.20:4438 , piece_id=JT76MOPCNNCSCXQIO4LLIXQI7H37AGAN2PU54HBRTK6SSTXQZXHQ, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.820374738 (291h51m42.371845199s) [noise=true , address=77.90.31.11:4553 , piece_id=EYA27FZI7ODACZHZ4VKQO6CVIUQIA2RFZAQO6SNQNMUF75JN4LKA, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:43.822855832 (291h51m42.374326293s) [noise=true , address=45.91.116.11:4119 , piece_id=5M5D2FQTS6W6YNJ7ZMD2PNCJFDVB6DUQS4W3I4HGPT4PNPFPZBZQ, seg_pos={0 0}, upload=257030]: Uploading piece... Nov 21 11:55:45.517678635 (291h51m44.069149096s) [upload=257030]: Next returned nil segment Nov 21 11:55:45.517698223 (291h51m44.069168684s) [upload=257030]: Waiting for error group managing segments... Nov 21 11:55:45.799866626 (291h51m44.351337087s) [noise=true , address=145.79.227.22:4509 , piece_id=JU7XTRZGA6UTOWQ2T7ELDVQ43LPVFZGJS4CHZHO5UKZQZOIHDL2A, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:45.810008537 (291h51m44.361479038s) [noise=true , address=145.79.227.11:4731 , piece_id=MHJZ3XCIOQSIBUGGBESKSB4YUNL2MRWF7KAB3DNOJXY4YAJUU4DQ, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:45.811011537 (291h51m44.362481998s) [noise=true , address=145.79.227.10:4401 , piece_id=SSMPPQJHJJREY43ZY6HHZTNPJ2AW4M2TTKQTTXJY72EZ53D4YHYQ, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:45.811704066 (291h51m44.363174527s) [noise=true , address=145.79.227.31:4797 , piece_id=IM7URIOGHOOBAQJH737TVDHC65ME35CVY2B7CQMTWNDML6J4TU2Q, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:45.813893355 (291h51m44.365363826s) [noise=true , address=145.79.227.16:4911 , piece_id=U7C7MJOGXDCKAGRLPHVHT4BLYD7X3QPJVHZAADXYA2P4PN2D56YA, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:45.814541068 (291h51m44.366011529s) [noise=true , address=145.79.227.23:4547 , piece_id=ZUBVHF47DA7YN4KMRW6JAEGKQ2B3ERPXCRKTYGGO3NBPI62VBAZQ, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:45.822110762 (291h51m44.373581223s) [noise=true , address=145.79.227.23:4546 , piece_id=RHP7FMWGYLV3DWZ67JSYRR4K74TCGZ756VRCCYWUX7OVZ57URZFA, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:45.829280465 (291h51m44.380750926s) [noise=true , address=145.79.227.17:4970 , piece_id=YTH4UCJPW4QY5L3KAIMGRUCRLWQDIVD3EUH5KR6UI6HJQ57VKUCA, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:45.832227386 (291h51m44.383697848s) [noise=true , address=145.79.227.27:4689 , piece_id=K64ZSMBDDY2YLEZNQR7R43CTNSEPV6EBFLOLHCUPWXKAU63EUBFA, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:45.833137700 (291h51m44.384608151s) [noise=true , address=145.79.227.20:4438 , piece_id=JT76MOPCNNCSCXQIO4LLIXQI7H37AGAN2PU54HBRTK6SSTXQZXHQ, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:45.838219727 (291h51m44.389690178s) [noise=true , address=145.79.227.21:4475 , piece_id=4KJ6F2VL2XYSM4LAVSLGI5G4YD47CPYCCF5D6VO6737T74WUHVVQ, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:46.265569192 (291h51m44.817039643s) [noise=true , address=77.90.31.13:4626 , piece_id=X7GQX3EDUFSPKGMWWI755MITJZEWIZSETYNII7WY6HZPSEVHSKMQ, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:46.277453250 (291h51m44.828923712s) [noise=true , address=45.91.116.13:4154 , piece_id=SFE5I7QNXOTRI6RUOFRSH3ROOGVT2XQYDKTXXM5AX7X7BVNCX7KQ, seg_pos={0 0}, upload=257030]: Done uploading piece. err: <nil> Nov 21 11:55:46.277483618 (291h51m44.828954079s) [seg_pos={0 0}, upload=257030]: Segment reached optimal threshold of 13 pieces. Nov 21 11:55:46.277733344 (291h51m44.829203805s) [noise=true , address=45.91.116.11:4119 , piece_id=5M5D2FQTS6W6YNJ7ZMD2PNCJFDVB6DUQS4W3I4HGPT4PNPFPZBZQ, seg_pos={0 0}, upload=257030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Nov 21 11:55:46.277744385 (291h51m44.829214837s) [noise=true , address=45.91.116.11:4119 , piece_id=5M5D2FQTS6W6YNJ7ZMD2PNCJFDVB6DUQS4W3I4HGPT4PNPFPZBZQ, seg_pos={0 0}, upload=257030]: Long tail cancellation due to optimal threshold Nov 21 11:55:46.278325271 (291h51m44.829795732s) [noise=true , address=77.90.31.17:4771 , piece_id=DEJCDBH4M4MXKYML5JNHHULV7M77VYROL5HYAZZKESLQY6USBL3Q, seg_pos={0 0}, upload=257030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Nov 21 11:55:46.278334860 (291h51m44.829805321s) [noise=true , address=77.90.31.17:4771 , piece_id=DEJCDBH4M4MXKYML5JNHHULV7M77VYROL5HYAZZKESLQY6USBL3Q, seg_pos={0 0}, upload=257030]: Long tail cancellation due to optimal threshold Nov 21 11:55:46.278485597 (291h51m44.829956048s) [noise=true , address=77.90.31.11:4553 , piece_id=EYA27FZI7ODACZHZ4VKQO6CVIUQIA2RFZAQO6SNQNMUF75JN4LKA, seg_pos={0 0}, upload=257030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Nov 21 11:55:46.278492039 (291h51m44.8299625s) [noise=true , address=77.90.31.11:4553 , piece_id=EYA27FZI7ODACZHZ4VKQO6CVIUQIA2RFZAQO6SNQNMUF75JN4LKA, seg_pos={0 0}, upload=257030]: Long tail cancellation due to optimal threshold Nov 21 11:55:46.278509352 (291h51m44.829979803s) [seg_pos={0 0}, upload=257030]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:55:46.278588763 (291h51m44.830059224s) [upload=257030]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:55:46.278593772 (291h51m44.830064234s) [upload=257030]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:55:46.568927373 (291h51m45.120397834s) [upload=257030]: Done uploading segments... Nov 21 11:55:46.568951348 (291h51m45.120421809s) [upload=257030]: Upload finished. err: <nil> Duration: 30.003899805s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........................] duration: 3.397914274s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:56:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.397947136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [............X.X.............................................] duration: 9.007567683s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:56:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 9.007595945s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.............X.X............................................] duration: 5.222538266s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:55:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.222565457s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X.XX...X.X.X....X............X....X.X.X..X.X..X...X..X] duration: 30.002178044s trace: https://jaeger.dev.storj.io/trace/17fb191e58216485 FAIL (2025-11-21 11:55:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 11:55:09.103277812 (291h51m7.654748273s) [upload=257020]: Starting upload Nov 21 11:55:09.103374266 (291h51m7.654844727s) [upload=257020]: Done starting upload Nov 21 11:55:09.103638109 (291h51m7.65510858s) [upload=257020]: Uploading segments... Nov 21 11:55:09.103688384 (291h51m7.655158855s) [upload=257020]: Got next segment. Inline: false Nov 21 11:55:09.103726086 (291h51m7.655196547s) [upload=257020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:55:09.103732348 (291h51m7.655202819s) [upload=257020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:55:09.350370453 (291h51m7.901840914s) [seg_pos={0 0}, upload=257020]: Begin upload segment... Nov 21 11:55:09.350431439 (291h51m7.90190191s) [seg_pos={0 0}, upload=257020]: Done begin upload segment. Nov 21 11:55:09.350463831 (291h51m7.901934292s) [noise=true , address=45.91.116.9:4263 , piece_id=N6I6E2HJYJ2DEUWBFDKUO4ADCBDIQ75KHZG5AJAGKM2A2FYRS4JA, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.350471044 (291h51m7.901941506s) [noise=true , address=145.79.227.26:4654 , piece_id=RLKPVLKP3SYQOWIFE5TCPOCOFHEO43J5YCQUD5BEODAD5SUCHDMQ, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.350942282 (291h51m7.902412743s) [noise=true , address=77.90.31.19:4841 , piece_id=LGBQ25ETXKKCNCL6GPONYNQ752DFUQPOFBELBW3S753L7TNG4X5Q, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.351017976 (291h51m7.902488437s) [noise=true , address=77.90.31.16:4734 , piece_id=7TY4GNT33VEWU4PBXDX54DEDWPEY2C7WLHUH2XURD6I3ZVWFDGMA, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.351446271 (291h51m7.902916732s) [noise=true , address=145.79.227.24:4581 , piece_id=2YHLTIJUHE6IABIV3ASCSEPGNZILCZEYEZMXNAPVT5G6OSHJNATQ, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.351872524 (291h51m7.903342985s) [noise=true , address=145.79.227.11:4730 , piece_id=BZX2PXWNWYQQ2QXZ6SMA7C4RO5MMFQGI5EJ4OH34ODKKN6DUWZ5A, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.352311840 (291h51m7.903782301s) [noise=true , address=77.90.31.17:4770 , piece_id=MY246IHCSPI4RQP3PEFUWATH6WSKVHAJREUB3RDVZJT3FXRM3LGA, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.352922453 (291h51m7.904392914s) [noise=true , address=145.79.227.20:4438 , piece_id=OPW7MDFQJMDLS4VGQXTB5QBVVZUXUS5QG4NAGMV6XFS3BPSDWIFA, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.353430149 (291h51m7.90490062s) [noise=true , address=145.79.227.27:4689 , piece_id=B635PQOXP4XNOANB3U4WVBKRG4W2ZUF2FOCBFSG3ONM5I2J32OZQ, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.354033519 (291h51m7.90550398s) [noise=true , address=145.79.227.21:4474 , piece_id=Z4YD57ZMYX5YXWHV3JZZVE4FKAJ25JGANXIHILKDYTMJJTUUH5UQ, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.354071631 (291h51m7.905542092s) [noise=true , address=145.79.227.15:4876 , piece_id=OKHRWHA3TPQ4I74HIRJB6H6HUZGJTF2NS2JJULKQTAKJSTWNDL6A, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.354539462 (291h51m7.906009923s) [noise=true , address=145.79.227.14:4838 , piece_id=OM5N3FFXNXQBZRKALKVT4WN3RTOUQOTRTVKSBOFYIYHVLI7UZSVA, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.354578355 (291h51m7.906048807s) [noise=true , address=145.79.227.15:4875 , piece_id=57DVREJ7P67IQABDL2VEZ57Q3D673Q6WAFMLNXDS4EGULNK7N4RQ, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.355194659 (291h51m7.90666512s) [noise=true , address=45.91.116.13:4154 , piece_id=HRKT4WH3WXRYKEAGBENLFWZ56A6YTV7OVCJXPM675VLNQ2OQPG5Q, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.355815953 (291h51m7.907286414s) [noise=true , address=77.90.31.13:4626 , piece_id=75NTZVNAOBWMFN4KDZI2DYA5PUFEWDWYJOMYEAITMY5JZRKX6SWA, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:09.356348176 (291h51m7.907818637s) [noise=true , address=145.79.227.16:4911 , piece_id=6TY7JAHQZ7XDSK65Y5CM4G64QN5Z5D6SKZA565FXGDJBTUVZ5M3A, seg_pos={0 0}, upload=257020]: Uploading piece... Nov 21 11:55:10.966139213 (291h51m9.517609674s) [upload=257020]: Next returned nil segment Nov 21 11:55:10.966198775 (291h51m9.517669256s) [upload=257020]: Waiting for error group managing segments... Nov 21 11:55:11.343606651 (291h51m9.895077112s) [noise=true , address=145.79.227.14:4838 , piece_id=OM5N3FFXNXQBZRKALKVT4WN3RTOUQOTRTVKSBOFYIYHVLI7UZSVA, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.348598527 (291h51m9.900068988s) [noise=true , address=145.79.227.26:4654 , piece_id=RLKPVLKP3SYQOWIFE5TCPOCOFHEO43J5YCQUD5BEODAD5SUCHDMQ, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.349138325 (291h51m9.900608776s) [noise=true , address=145.79.227.24:4581 , piece_id=2YHLTIJUHE6IABIV3ASCSEPGNZILCZEYEZMXNAPVT5G6OSHJNATQ, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.349411866 (291h51m9.900882327s) [noise=true , address=145.79.227.15:4876 , piece_id=OKHRWHA3TPQ4I74HIRJB6H6HUZGJTF2NS2JJULKQTAKJSTWNDL6A, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.351205862 (291h51m9.902676323s) [noise=true , address=145.79.227.20:4438 , piece_id=OPW7MDFQJMDLS4VGQXTB5QBVVZUXUS5QG4NAGMV6XFS3BPSDWIFA, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.351468231 (291h51m9.902938692s) [noise=true , address=145.79.227.11:4730 , piece_id=BZX2PXWNWYQQ2QXZ6SMA7C4RO5MMFQGI5EJ4OH34ODKKN6DUWZ5A, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.351914050 (291h51m9.903384521s) [noise=true , address=145.79.227.21:4474 , piece_id=Z4YD57ZMYX5YXWHV3JZZVE4FKAJ25JGANXIHILKDYTMJJTUUH5UQ, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.359782143 (291h51m9.911252604s) [noise=true , address=145.79.227.15:4875 , piece_id=57DVREJ7P67IQABDL2VEZ57Q3D673Q6WAFMLNXDS4EGULNK7N4RQ, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.374181512 (291h51m9.925651973s) [noise=true , address=145.79.227.27:4689 , piece_id=B635PQOXP4XNOANB3U4WVBKRG4W2ZUF2FOCBFSG3ONM5I2J32OZQ, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.374542750 (291h51m9.926013211s) [noise=true , address=145.79.227.16:4911 , piece_id=6TY7JAHQZ7XDSK65Y5CM4G64QN5Z5D6SKZA565FXGDJBTUVZ5M3A, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.798276609 (291h51m10.34974707s) [noise=true , address=77.90.31.13:4626 , piece_id=75NTZVNAOBWMFN4KDZI2DYA5PUFEWDWYJOMYEAITMY5JZRKX6SWA, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.840618818 (291h51m10.392089279s) [noise=true , address=77.90.31.19:4841 , piece_id=LGBQ25ETXKKCNCL6GPONYNQ752DFUQPOFBELBW3S753L7TNG4X5Q, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.864411372 (291h51m10.415881833s) [noise=true , address=45.91.116.13:4154 , piece_id=HRKT4WH3WXRYKEAGBENLFWZ56A6YTV7OVCJXPM675VLNQ2OQPG5Q, seg_pos={0 0}, upload=257020]: Done uploading piece. err: <nil> Nov 21 11:55:11.864437833 (291h51m10.415908294s) [seg_pos={0 0}, upload=257020]: Segment reached optimal threshold of 13 pieces. Nov 21 11:55:11.864655177 (291h51m10.416125638s) [noise=true , address=77.90.31.16:4734 , piece_id=7TY4GNT33VEWU4PBXDX54DEDWPEY2C7WLHUH2XURD6I3ZVWFDGMA, seg_pos={0 0}, upload=257020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Nov 21 11:55:11.864664615 (291h51m10.416135066s) [noise=true , address=77.90.31.16:4734 , piece_id=7TY4GNT33VEWU4PBXDX54DEDWPEY2C7WLHUH2XURD6I3ZVWFDGMA, seg_pos={0 0}, upload=257020]: Long tail cancellation due to optimal threshold Nov 21 11:55:11.864777340 (291h51m10.416247811s) [noise=true , address=45.91.116.9:4263 , piece_id=N6I6E2HJYJ2DEUWBFDKUO4ADCBDIQ75KHZG5AJAGKM2A2FYRS4JA, seg_pos={0 0}, upload=257020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:57270->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:57270->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57270->45.91.116.9:4263: use of closed network connection Nov 21 11:55:11.864789623 (291h51m10.416260084s) [noise=true , address=45.91.116.9:4263 , piece_id=N6I6E2HJYJ2DEUWBFDKUO4ADCBDIQ75KHZG5AJAGKM2A2FYRS4JA, seg_pos={0 0}, upload=257020]: Long tail cancellation due to optimal threshold Nov 21 11:55:11.864924239 (291h51m10.4163947s) [noise=true , address=77.90.31.17:4770 , piece_id=MY246IHCSPI4RQP3PEFUWATH6WSKVHAJREUB3RDVZJT3FXRM3LGA, seg_pos={0 0}, upload=257020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 11:55:11.864931483 (291h51m10.416401944s) [noise=true , address=77.90.31.17:4770 , piece_id=MY246IHCSPI4RQP3PEFUWATH6WSKVHAJREUB3RDVZJT3FXRM3LGA, seg_pos={0 0}, upload=257020]: Long tail cancellation due to optimal threshold Nov 21 11:55:11.864950740 (291h51m10.416421191s) [seg_pos={0 0}, upload=257020]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:55:11.865022076 (291h51m10.416492527s) [upload=257020]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:55:11.865027666 (291h51m10.416498127s) [upload=257020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:55:12.162776417 (291h51m10.714246888s) [upload=257020]: Done uploading segments... Nov 21 11:55:12.162801184 (291h51m10.714271645s) [upload=257020]: Upload finished. err: <nil> Duration: 30.002201669s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......................] duration: 3.418847514s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:54:57 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.418879606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 5.39053402s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:54:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.390563064s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......................] duration: 3.677844844s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:54:37 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.677873609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X......X.XX...X.X.X....X............X....X.X.X..X.X..X...X..] duration: 6.431787563s trace: https://jaeger.dev.storj.io/trace/513850953a062c9b FAIL (2025-11-21 11:54:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA) context deadline exceeded ======LOGS====== 21 11:52:18.550541569 (291h48m17.10201203s) [upload=256978]: Starting upload Nov 21 11:52:18.550634156 (291h48m17.102104617s) [upload=256978]: Done starting upload Nov 21 11:52:18.550903869 (291h48m17.10237433s) [upload=256978]: Uploading segments... Nov 21 11:52:18.550955708 (291h48m17.102426169s) [upload=256978]: Got next segment. Inline: false Nov 21 11:52:18.550997727 (291h48m17.102468189s) [upload=256978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:52:18.551003969 (291h48m17.10247443s) [upload=256978]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:52:18.796351348 (291h48m17.347821809s) [seg_pos={0 0}, upload=256978]: Begin upload segment... Nov 21 11:52:18.796415901 (291h48m17.347886352s) [seg_pos={0 0}, upload=256978]: Done begin upload segment. Nov 21 11:52:18.796454224 (291h48m17.347924685s) [noise=true , address=145.79.227.17:4971 , piece_id=SU7OAQ3IQDYLVINT3UNK6M7YDVWGI4DJMXZXXD2AEMKEXPJ4RRFA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.796581256 (291h48m17.348051717s) [noise=true , address=145.79.227.24:4583 , piece_id=YHFQLMSJJ3XMUWK7OJ2247WGMEE3EV5DMPNZNIA5GD7P23K5VITA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797167943 (291h48m17.348638394s) [noise=true , address=145.79.227.32:4835 , piece_id=VLPQKL7A6II7Q3573BCUIP3YQAMKRTD6I2MKWR55SKTUZ6P2XRHQ, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797249749 (291h48m17.34872021s) [noise=true , address=145.79.227.21:4475 , piece_id=5UJ5JQORO24UGNVTZ4GWHGQIN7Z4YZ2DSQBBJPZTIFWGRAGTPCSA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797609514 (291h48m17.349079976s) [noise=true , address=145.79.227.30:4762 , piece_id=5A35DXNMNHWWWLA7CLOPJK7VJ4HEPFZ62H6AAQAO3HC5FSQPO7NA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797681652 (291h48m17.349152113s) [noise=true , address=145.79.227.9:4366 , piece_id=XAP3WCJMA7S4LMDQP6FF5QUXR2C475RKPHXZNWUKFV6WWNFE5B5A, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798011419 (291h48m17.34948187s) [noise=true , address=145.79.227.26:4653 , piece_id=B7UUQOEQLBS4BQGXFT4S5WAADKS7GHIRQ347V74EIEGBFAW2GDQA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798157647 (291h48m17.349628099s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798423614 (291h48m17.349894065s) [noise=true , address=77.90.31.12:4592 , piece_id=ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798586044 (291h48m17.350056505s) [noise=true , address=145.79.227.8:4329 , piece_id=OO4DEPELLYK2FTBHKTKGXZKWYURJVDM2XP3ML4TUWC7OC7M3WW5Q, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798812745 (291h48m17.350283196s) [noise=true , address=145.79.227.13:4801 , piece_id=RMIFD5T33USCOVYIBRFHK5T6JRKAPDTESARJBC7AJ3I6AXGQFDWQ, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799017114 (291h48m17.350487575s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799177530 (291h48m17.350647991s) [noise=true , address=145.79.227.10:4402 , piece_id=XTYNEHERZ37JUMFEQ24HCXWY3T4EU3EDID6Q3AHTIDGW6CW4OTSA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799472952 (291h48m17.350943413s) [noise=true , address=45.91.116.14:4191 , piece_id=QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799656962 (291h48m17.351127413s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799861031 (291h48m17.351331502s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:20.405914941 (291h48m18.957385402s) [upload=256978]: Next returned nil segment Nov 21 11:52:20.405946161 (291h48m18.957416622s) [upload=256978]: Waiting for error group managing segments... Nov 21 11:52:20.788637197 (291h48m19.340107658s) [noise=true , address=145.79.227.24:4583 , piece_id=YHFQLMSJJ3XMUWK7OJ2247WGMEE3EV5DMPNZNIA5GD7P23K5VITA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.792094981 (291h48m19.343565432s) [noise=true , address=145.79.227.21:4475 , piece_id=5UJ5JQORO24UGNVTZ4GWHGQIN7Z4YZ2DSQBBJPZTIFWGRAGTPCSA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.793185819 (291h48m19.34465628s) [noise=true , address=145.79.227.13:4801 , piece_id=RMIFD5T33USCOVYIBRFHK5T6JRKAPDTESARJBC7AJ3I6AXGQFDWQ, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.795089112 (291h48m19.346559573s) [noise=true , address=145.79.227.9:4366 , piece_id=XAP3WCJMA7S4LMDQP6FF5QUXR2C475RKPHXZNWUKFV6WWNFE5B5A, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.795745082 (291h48m19.347215543s) [noise=true , address=145.79.227.17:4971 , piece_id=SU7OAQ3IQDYLVINT3UNK6M7YDVWGI4DJMXZXXD2AEMKEXPJ4RRFA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.798623102 (291h48m19.350093563s) [noise=true , address=145.79.227.30:4762 , piece_id=5A35DXNMNHWWWLA7CLOPJK7VJ4HEPFZ62H6AAQAO3HC5FSQPO7NA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.809441713 (291h48m19.360912174s) [noise=true , address=145.79.227.10:4402 , piece_id=XTYNEHERZ37JUMFEQ24HCXWY3T4EU3EDID6Q3AHTIDGW6CW4OTSA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.809974137 (291h48m19.361444588s) [noise=true , address=145.79.227.8:4329 , piece_id=OO4DEPELLYK2FTBHKTKGXZKWYURJVDM2XP3ML4TUWC7OC7M3WW5Q, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.813153782 (291h48m19.364624233s) [noise=true , address=145.79.227.32:4835 , piece_id=VLPQKL7A6II7Q3573BCUIP3YQAMKRTD6I2MKWR55SKTUZ6P2XRHQ, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.813743304 (291h48m19.365213765s) [noise=true , address=145.79.227.26:4653 , piece_id=B7UUQOEQLBS4BQGXFT4S5WAADKS7GHIRQ347V74EIEGBFAW2GDQA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:21.328226067 (291h48m19.879696538s) [noise=true , address=45.91.116.14:4191 , piece_id=QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:21.416925216 (291h48m19.968395667s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:22.476743402 (291h48m21.028213863s) [noise=true , address=77.90.31.12:4592 , piece_id=ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:22.476768199 (291h48m21.02823866s) [seg_pos={0 0}, upload=256978]: Segment reached optimal threshold of 13 pieces. Nov 21 11:52:22.477024557 (291h48m21.028495018s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Nov 21 11:52:22.477036120 (291h48m21.028506581s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477064013 (291h48m21.028534484s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 11:52:22.477074463 (291h48m21.028544924s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477289833 (291h48m21.028760284s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection Nov 21 11:52:22.477299491 (291h48m21.028769962s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477330400 (291h48m21.028800851s) [seg_pos={0 0}, upload=256978]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:52:22.477448535 (291h48m21.028918996s) [upload=256978]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:52:22.477465617 (291h48m21.028936088s) [upload=256978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:52:22.771779587 (291h48m21.323250058s) [upload=256978]: Done uploading segments... Nov 21 11:52:22.771808563 (291h48m21.323279014s) [upload=256978]: Upload finished. err: <nil> Duration: 6.431904456s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 5.906185629s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:53:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.9062126s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....................] duration: 3.728177739s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:53:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.728199661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 6.687670494s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:53:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.687702463s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....................] duration: 4.061749811s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:53:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.061778276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X......X.XX...X.X.X....X............X....X.X.X..X.X..X...X.] duration: 5.781521843s trace: https://jaeger.dev.storj.io/trace/513850953a062c9b FAIL (2025-11-21 11:52:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA) context deadline exceeded ======LOGS====== 21 11:52:18.550541569 (291h48m17.10201203s) [upload=256978]: Starting upload Nov 21 11:52:18.550634156 (291h48m17.102104617s) [upload=256978]: Done starting upload Nov 21 11:52:18.550903869 (291h48m17.10237433s) [upload=256978]: Uploading segments... Nov 21 11:52:18.550955708 (291h48m17.102426169s) [upload=256978]: Got next segment. Inline: false Nov 21 11:52:18.550997727 (291h48m17.102468189s) [upload=256978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:52:18.551003969 (291h48m17.10247443s) [upload=256978]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:52:18.796351348 (291h48m17.347821809s) [seg_pos={0 0}, upload=256978]: Begin upload segment... Nov 21 11:52:18.796415901 (291h48m17.347886352s) [seg_pos={0 0}, upload=256978]: Done begin upload segment. Nov 21 11:52:18.796454224 (291h48m17.347924685s) [noise=true , address=145.79.227.17:4971 , piece_id=SU7OAQ3IQDYLVINT3UNK6M7YDVWGI4DJMXZXXD2AEMKEXPJ4RRFA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.796581256 (291h48m17.348051717s) [noise=true , address=145.79.227.24:4583 , piece_id=YHFQLMSJJ3XMUWK7OJ2247WGMEE3EV5DMPNZNIA5GD7P23K5VITA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797167943 (291h48m17.348638394s) [noise=true , address=145.79.227.32:4835 , piece_id=VLPQKL7A6II7Q3573BCUIP3YQAMKRTD6I2MKWR55SKTUZ6P2XRHQ, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797249749 (291h48m17.34872021s) [noise=true , address=145.79.227.21:4475 , piece_id=5UJ5JQORO24UGNVTZ4GWHGQIN7Z4YZ2DSQBBJPZTIFWGRAGTPCSA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797609514 (291h48m17.349079976s) [noise=true , address=145.79.227.30:4762 , piece_id=5A35DXNMNHWWWLA7CLOPJK7VJ4HEPFZ62H6AAQAO3HC5FSQPO7NA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797681652 (291h48m17.349152113s) [noise=true , address=145.79.227.9:4366 , piece_id=XAP3WCJMA7S4LMDQP6FF5QUXR2C475RKPHXZNWUKFV6WWNFE5B5A, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798011419 (291h48m17.34948187s) [noise=true , address=145.79.227.26:4653 , piece_id=B7UUQOEQLBS4BQGXFT4S5WAADKS7GHIRQ347V74EIEGBFAW2GDQA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798157647 (291h48m17.349628099s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798423614 (291h48m17.349894065s) [noise=true , address=77.90.31.12:4592 , piece_id=ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798586044 (291h48m17.350056505s) [noise=true , address=145.79.227.8:4329 , piece_id=OO4DEPELLYK2FTBHKTKGXZKWYURJVDM2XP3ML4TUWC7OC7M3WW5Q, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798812745 (291h48m17.350283196s) [noise=true , address=145.79.227.13:4801 , piece_id=RMIFD5T33USCOVYIBRFHK5T6JRKAPDTESARJBC7AJ3I6AXGQFDWQ, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799017114 (291h48m17.350487575s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799177530 (291h48m17.350647991s) [noise=true , address=145.79.227.10:4402 , piece_id=XTYNEHERZ37JUMFEQ24HCXWY3T4EU3EDID6Q3AHTIDGW6CW4OTSA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799472952 (291h48m17.350943413s) [noise=true , address=45.91.116.14:4191 , piece_id=QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799656962 (291h48m17.351127413s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799861031 (291h48m17.351331502s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:20.405914941 (291h48m18.957385402s) [upload=256978]: Next returned nil segment Nov 21 11:52:20.405946161 (291h48m18.957416622s) [upload=256978]: Waiting for error group managing segments... Nov 21 11:52:20.788637197 (291h48m19.340107658s) [noise=true , address=145.79.227.24:4583 , piece_id=YHFQLMSJJ3XMUWK7OJ2247WGMEE3EV5DMPNZNIA5GD7P23K5VITA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.792094981 (291h48m19.343565432s) [noise=true , address=145.79.227.21:4475 , piece_id=5UJ5JQORO24UGNVTZ4GWHGQIN7Z4YZ2DSQBBJPZTIFWGRAGTPCSA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.793185819 (291h48m19.34465628s) [noise=true , address=145.79.227.13:4801 , piece_id=RMIFD5T33USCOVYIBRFHK5T6JRKAPDTESARJBC7AJ3I6AXGQFDWQ, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.795089112 (291h48m19.346559573s) [noise=true , address=145.79.227.9:4366 , piece_id=XAP3WCJMA7S4LMDQP6FF5QUXR2C475RKPHXZNWUKFV6WWNFE5B5A, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.795745082 (291h48m19.347215543s) [noise=true , address=145.79.227.17:4971 , piece_id=SU7OAQ3IQDYLVINT3UNK6M7YDVWGI4DJMXZXXD2AEMKEXPJ4RRFA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.798623102 (291h48m19.350093563s) [noise=true , address=145.79.227.30:4762 , piece_id=5A35DXNMNHWWWLA7CLOPJK7VJ4HEPFZ62H6AAQAO3HC5FSQPO7NA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.809441713 (291h48m19.360912174s) [noise=true , address=145.79.227.10:4402 , piece_id=XTYNEHERZ37JUMFEQ24HCXWY3T4EU3EDID6Q3AHTIDGW6CW4OTSA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.809974137 (291h48m19.361444588s) [noise=true , address=145.79.227.8:4329 , piece_id=OO4DEPELLYK2FTBHKTKGXZKWYURJVDM2XP3ML4TUWC7OC7M3WW5Q, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.813153782 (291h48m19.364624233s) [noise=true , address=145.79.227.32:4835 , piece_id=VLPQKL7A6II7Q3573BCUIP3YQAMKRTD6I2MKWR55SKTUZ6P2XRHQ, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.813743304 (291h48m19.365213765s) [noise=true , address=145.79.227.26:4653 , piece_id=B7UUQOEQLBS4BQGXFT4S5WAADKS7GHIRQ347V74EIEGBFAW2GDQA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:21.328226067 (291h48m19.879696538s) [noise=true , address=45.91.116.14:4191 , piece_id=QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:21.416925216 (291h48m19.968395667s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:22.476743402 (291h48m21.028213863s) [noise=true , address=77.90.31.12:4592 , piece_id=ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:22.476768199 (291h48m21.02823866s) [seg_pos={0 0}, upload=256978]: Segment reached optimal threshold of 13 pieces. Nov 21 11:52:22.477024557 (291h48m21.028495018s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Nov 21 11:52:22.477036120 (291h48m21.028506581s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477064013 (291h48m21.028534484s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 11:52:22.477074463 (291h48m21.028544924s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477289833 (291h48m21.028760284s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection Nov 21 11:52:22.477299491 (291h48m21.028769962s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477330400 (291h48m21.028800851s) [seg_pos={0 0}, upload=256978]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:52:22.477448535 (291h48m21.028918996s) [upload=256978]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:52:22.477465617 (291h48m21.028936088s) [upload=256978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:52:22.771779587 (291h48m21.323250058s) [upload=256978]: Done uploading segments... Nov 21 11:52:22.771808563 (291h48m21.323279014s) [upload=256978]: Upload finished. err: <nil> Duration: 5.781541352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X......X.XX...X.X.X....X............X....X.X.X..X.X..X...X] duration: 30.002020265s trace: https://jaeger.dev.storj.io/trace/513850953a062c9b FAIL (2025-11-21 11:52:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA) context deadline exceeded; piecestore: (Node ID: 12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, Piece ID: ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA) context deadline exceeded ======LOGS====== 21 11:52:18.550541569 (291h48m17.10201203s) [upload=256978]: Starting upload Nov 21 11:52:18.550634156 (291h48m17.102104617s) [upload=256978]: Done starting upload Nov 21 11:52:18.550903869 (291h48m17.10237433s) [upload=256978]: Uploading segments... Nov 21 11:52:18.550955708 (291h48m17.102426169s) [upload=256978]: Got next segment. Inline: false Nov 21 11:52:18.550997727 (291h48m17.102468189s) [upload=256978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:52:18.551003969 (291h48m17.10247443s) [upload=256978]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:52:18.796351348 (291h48m17.347821809s) [seg_pos={0 0}, upload=256978]: Begin upload segment... Nov 21 11:52:18.796415901 (291h48m17.347886352s) [seg_pos={0 0}, upload=256978]: Done begin upload segment. Nov 21 11:52:18.796454224 (291h48m17.347924685s) [noise=true , address=145.79.227.17:4971 , piece_id=SU7OAQ3IQDYLVINT3UNK6M7YDVWGI4DJMXZXXD2AEMKEXPJ4RRFA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.796581256 (291h48m17.348051717s) [noise=true , address=145.79.227.24:4583 , piece_id=YHFQLMSJJ3XMUWK7OJ2247WGMEE3EV5DMPNZNIA5GD7P23K5VITA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797167943 (291h48m17.348638394s) [noise=true , address=145.79.227.32:4835 , piece_id=VLPQKL7A6II7Q3573BCUIP3YQAMKRTD6I2MKWR55SKTUZ6P2XRHQ, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797249749 (291h48m17.34872021s) [noise=true , address=145.79.227.21:4475 , piece_id=5UJ5JQORO24UGNVTZ4GWHGQIN7Z4YZ2DSQBBJPZTIFWGRAGTPCSA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797609514 (291h48m17.349079976s) [noise=true , address=145.79.227.30:4762 , piece_id=5A35DXNMNHWWWLA7CLOPJK7VJ4HEPFZ62H6AAQAO3HC5FSQPO7NA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.797681652 (291h48m17.349152113s) [noise=true , address=145.79.227.9:4366 , piece_id=XAP3WCJMA7S4LMDQP6FF5QUXR2C475RKPHXZNWUKFV6WWNFE5B5A, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798011419 (291h48m17.34948187s) [noise=true , address=145.79.227.26:4653 , piece_id=B7UUQOEQLBS4BQGXFT4S5WAADKS7GHIRQ347V74EIEGBFAW2GDQA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798157647 (291h48m17.349628099s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798423614 (291h48m17.349894065s) [noise=true , address=77.90.31.12:4592 , piece_id=ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798586044 (291h48m17.350056505s) [noise=true , address=145.79.227.8:4329 , piece_id=OO4DEPELLYK2FTBHKTKGXZKWYURJVDM2XP3ML4TUWC7OC7M3WW5Q, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.798812745 (291h48m17.350283196s) [noise=true , address=145.79.227.13:4801 , piece_id=RMIFD5T33USCOVYIBRFHK5T6JRKAPDTESARJBC7AJ3I6AXGQFDWQ, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799017114 (291h48m17.350487575s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799177530 (291h48m17.350647991s) [noise=true , address=145.79.227.10:4402 , piece_id=XTYNEHERZ37JUMFEQ24HCXWY3T4EU3EDID6Q3AHTIDGW6CW4OTSA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799472952 (291h48m17.350943413s) [noise=true , address=45.91.116.14:4191 , piece_id=QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799656962 (291h48m17.351127413s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:18.799861031 (291h48m17.351331502s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Uploading piece... Nov 21 11:52:20.405914941 (291h48m18.957385402s) [upload=256978]: Next returned nil segment Nov 21 11:52:20.405946161 (291h48m18.957416622s) [upload=256978]: Waiting for error group managing segments... Nov 21 11:52:20.788637197 (291h48m19.340107658s) [noise=true , address=145.79.227.24:4583 , piece_id=YHFQLMSJJ3XMUWK7OJ2247WGMEE3EV5DMPNZNIA5GD7P23K5VITA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.792094981 (291h48m19.343565432s) [noise=true , address=145.79.227.21:4475 , piece_id=5UJ5JQORO24UGNVTZ4GWHGQIN7Z4YZ2DSQBBJPZTIFWGRAGTPCSA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.793185819 (291h48m19.34465628s) [noise=true , address=145.79.227.13:4801 , piece_id=RMIFD5T33USCOVYIBRFHK5T6JRKAPDTESARJBC7AJ3I6AXGQFDWQ, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.795089112 (291h48m19.346559573s) [noise=true , address=145.79.227.9:4366 , piece_id=XAP3WCJMA7S4LMDQP6FF5QUXR2C475RKPHXZNWUKFV6WWNFE5B5A, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.795745082 (291h48m19.347215543s) [noise=true , address=145.79.227.17:4971 , piece_id=SU7OAQ3IQDYLVINT3UNK6M7YDVWGI4DJMXZXXD2AEMKEXPJ4RRFA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.798623102 (291h48m19.350093563s) [noise=true , address=145.79.227.30:4762 , piece_id=5A35DXNMNHWWWLA7CLOPJK7VJ4HEPFZ62H6AAQAO3HC5FSQPO7NA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.809441713 (291h48m19.360912174s) [noise=true , address=145.79.227.10:4402 , piece_id=XTYNEHERZ37JUMFEQ24HCXWY3T4EU3EDID6Q3AHTIDGW6CW4OTSA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.809974137 (291h48m19.361444588s) [noise=true , address=145.79.227.8:4329 , piece_id=OO4DEPELLYK2FTBHKTKGXZKWYURJVDM2XP3ML4TUWC7OC7M3WW5Q, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.813153782 (291h48m19.364624233s) [noise=true , address=145.79.227.32:4835 , piece_id=VLPQKL7A6II7Q3573BCUIP3YQAMKRTD6I2MKWR55SKTUZ6P2XRHQ, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:20.813743304 (291h48m19.365213765s) [noise=true , address=145.79.227.26:4653 , piece_id=B7UUQOEQLBS4BQGXFT4S5WAADKS7GHIRQ347V74EIEGBFAW2GDQA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:21.328226067 (291h48m19.879696538s) [noise=true , address=45.91.116.14:4191 , piece_id=QTNPHDNVVEYPLYUQ65RHE6XTRGLTDEIZ3YABFKUGGYZPREECM2RA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:21.416925216 (291h48m19.968395667s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJOXQDH4O3QTH5UTFYPKWTRSJPVQWRB3EZOW3QAOZKG5NYGAFQA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:22.476743402 (291h48m21.028213863s) [noise=true , address=77.90.31.12:4592 , piece_id=ISEEO47BNDEDRTVBYWQNNLRLW3UCKETKU6YOB3U2QBT6765GZMZA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: <nil> Nov 21 11:52:22.476768199 (291h48m21.02823866s) [seg_pos={0 0}, upload=256978]: Segment reached optimal threshold of 13 pieces. Nov 21 11:52:22.477024557 (291h48m21.028495018s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Nov 21 11:52:22.477036120 (291h48m21.028506581s) [noise=true , address=77.90.31.19:4843 , piece_id=GL5JTR6KPVN3MXWTVXXAIT26DCRSVTDDC74XOJBK75Y5XY2P2RFA, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477064013 (291h48m21.028534484s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 11:52:22.477074463 (291h48m21.028544924s) [noise=true , address=77.90.31.17:4770 , piece_id=ZXV3HX7U6ITP6HA7H2KIEDBBWZBWAIA2XDJ3C3IWXWIU5YGGGA3Q, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477289833 (291h48m21.028760284s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52312->45.91.116.11:4117: use of closed network connection Nov 21 11:52:22.477299491 (291h48m21.028769962s) [noise=true , address=45.91.116.11:4117 , piece_id=R52CHGV4Z4777BMJ3HUHCJO5COC27VT2LHL34EDD2SYO5CTYNCRA, seg_pos={0 0}, upload=256978]: Long tail cancellation due to optimal threshold Nov 21 11:52:22.477330400 (291h48m21.028800851s) [seg_pos={0 0}, upload=256978]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:52:22.477448535 (291h48m21.028918996s) [upload=256978]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:52:22.477465617 (291h48m21.028936088s) [upload=256978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:52:22.771779587 (291h48m21.323250058s) [upload=256978]: Done uploading segments... Nov 21 11:52:22.771808563 (291h48m21.323279014s) [upload=256978]: Upload finished. err: <nil> Duration: 30.002047107s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...................] duration: 3.653921829s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:52:21 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.653961384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 5.308869665s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:52:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.308895965s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 6.00046312s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:52:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.000496994s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X......X.XX...X.X.X....X............X....X.X.X..X.X..X...] duration: 6.343416669s trace: https://jaeger.dev.storj.io/trace/34d009b2527f9379 FAIL (2025-11-21 11:51:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ) context deadline exceeded ======LOGS====== ments... Nov 21 11:49:16.994612805 (291h45m15.546083266s) [upload=256935]: Got next segment. Inline: false Nov 21 11:49:16.994647011 (291h45m15.546117472s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:49:16.994651759 (291h45m15.54612221s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:49:17.243148153 (291h45m15.794618604s) [seg_pos={0 0}, upload=256935]: Begin upload segment... Nov 21 11:49:17.243197467 (291h45m15.794667929s) [seg_pos={0 0}, upload=256935]: Done begin upload segment. Nov 21 11:49:17.243223908 (291h45m15.794694369s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243288660 (291h45m15.794759112s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243999364 (291h45m15.795469815s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.244784930 (291h45m15.796255391s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245522555 (291h45m15.796993016s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245623898 (291h45m15.797094359s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246035040 (291h45m15.797505491s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246184175 (291h45m15.797654636s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246797051 (291h45m15.798267522s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247334054 (291h45m15.798804515s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247548422 (291h45m15.799018873s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249260893 (291h45m15.800731354s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249559843 (291h45m15.801030304s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.251617881 (291h45m15.803088333s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.252868232 (291h45m15.804338704s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.254615760 (291h45m15.806086221s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:18.965747830 (291h45m17.517218281s) [upload=256935]: Next returned nil segment Nov 21 11:49:18.965769751 (291h45m17.517240243s) [upload=256935]: Waiting for error group managing segments... Nov 21 11:49:19.250399321 (291h45m17.801869792s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.250854327 (291h45m17.802324788s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.260959880 (291h45m17.812430341s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.262082247 (291h45m17.813552709s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.268139792 (291h45m17.819610263s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.280973680 (291h45m17.832444141s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.282868778 (291h45m17.83433924s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.286078351 (291h45m17.837548812s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.821632116 (291h45m18.373102577s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.824423853 (291h45m18.375894304s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.850707311 (291h45m18.402177772s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.908423644 (291h45m18.459894105s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958255192 (291h45m18.509725653s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958317902 (291h45m18.509788353s) [seg_pos={0 0}, upload=256935]: Segment reached optimal threshold of 13 pieces. Nov 21 11:49:19.958597444 (291h45m18.510067905s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection Nov 21 11:49:19.958607714 (291h45m18.510078175s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958655214 (291h45m18.510125675s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection Nov 21 11:49:19.958672887 (291h45m18.510143349s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958915429 (291h45m18.5103859s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Nov 21 11:49:19.958928194 (291h45m18.510398655s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958967489 (291h45m18.51043795s) [seg_pos={0 0}, upload=256935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:49:19.959064374 (291h45m18.510534835s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:49:19.959071227 (291h45m18.510541688s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:49:20.260011595 (291h45m18.811482076s) [upload=256935]: Done uploading segments... Nov 21 11:49:20.260040470 (291h45m18.811510931s) [upload=256935]: Upload finished. err: <nil> Duration: 6.343441355s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 5.287695967s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:51:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.287723541s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..................] duration: 3.603489297s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:51:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.603515297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.X......X.XX...X.X.X....X............X....X.X.X..X.X..X..] duration: 6.330563932s trace: https://jaeger.dev.storj.io/trace/34d009b2527f9379 FAIL (2025-11-21 11:50:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ) context deadline exceeded ======LOGS====== ments... Nov 21 11:49:16.994612805 (291h45m15.546083266s) [upload=256935]: Got next segment. Inline: false Nov 21 11:49:16.994647011 (291h45m15.546117472s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:49:16.994651759 (291h45m15.54612221s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:49:17.243148153 (291h45m15.794618604s) [seg_pos={0 0}, upload=256935]: Begin upload segment... Nov 21 11:49:17.243197467 (291h45m15.794667929s) [seg_pos={0 0}, upload=256935]: Done begin upload segment. Nov 21 11:49:17.243223908 (291h45m15.794694369s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243288660 (291h45m15.794759112s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243999364 (291h45m15.795469815s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.244784930 (291h45m15.796255391s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245522555 (291h45m15.796993016s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245623898 (291h45m15.797094359s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246035040 (291h45m15.797505491s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246184175 (291h45m15.797654636s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246797051 (291h45m15.798267522s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247334054 (291h45m15.798804515s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247548422 (291h45m15.799018873s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249260893 (291h45m15.800731354s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249559843 (291h45m15.801030304s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.251617881 (291h45m15.803088333s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.252868232 (291h45m15.804338704s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.254615760 (291h45m15.806086221s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:18.965747830 (291h45m17.517218281s) [upload=256935]: Next returned nil segment Nov 21 11:49:18.965769751 (291h45m17.517240243s) [upload=256935]: Waiting for error group managing segments... Nov 21 11:49:19.250399321 (291h45m17.801869792s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.250854327 (291h45m17.802324788s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.260959880 (291h45m17.812430341s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.262082247 (291h45m17.813552709s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.268139792 (291h45m17.819610263s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.280973680 (291h45m17.832444141s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.282868778 (291h45m17.83433924s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.286078351 (291h45m17.837548812s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.821632116 (291h45m18.373102577s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.824423853 (291h45m18.375894304s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.850707311 (291h45m18.402177772s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.908423644 (291h45m18.459894105s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958255192 (291h45m18.509725653s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958317902 (291h45m18.509788353s) [seg_pos={0 0}, upload=256935]: Segment reached optimal threshold of 13 pieces. Nov 21 11:49:19.958597444 (291h45m18.510067905s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection Nov 21 11:49:19.958607714 (291h45m18.510078175s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958655214 (291h45m18.510125675s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection Nov 21 11:49:19.958672887 (291h45m18.510143349s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958915429 (291h45m18.5103859s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Nov 21 11:49:19.958928194 (291h45m18.510398655s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958967489 (291h45m18.51043795s) [seg_pos={0 0}, upload=256935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:49:19.959064374 (291h45m18.510534835s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:49:19.959071227 (291h45m18.510541688s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:49:20.260011595 (291h45m18.811482076s) [upload=256935]: Done uploading segments... Nov 21 11:49:20.260040470 (291h45m18.811510931s) [upload=256935]: Upload finished. err: <nil> Duration: 6.33059497s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 7.271589155s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:50:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 7.271618511s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.................] duration: 3.539947932s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:50:35 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.53997371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.X......X.XX...X.X.X....X............X....X.X.X..X.X..X.] duration: 6.375072317s trace: https://jaeger.dev.storj.io/trace/34d009b2527f9379 FAIL (2025-11-21 11:50:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ) context deadline exceeded ======LOGS====== ments... Nov 21 11:49:16.994612805 (291h45m15.546083266s) [upload=256935]: Got next segment. Inline: false Nov 21 11:49:16.994647011 (291h45m15.546117472s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:49:16.994651759 (291h45m15.54612221s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:49:17.243148153 (291h45m15.794618604s) [seg_pos={0 0}, upload=256935]: Begin upload segment... Nov 21 11:49:17.243197467 (291h45m15.794667929s) [seg_pos={0 0}, upload=256935]: Done begin upload segment. Nov 21 11:49:17.243223908 (291h45m15.794694369s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243288660 (291h45m15.794759112s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243999364 (291h45m15.795469815s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.244784930 (291h45m15.796255391s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245522555 (291h45m15.796993016s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245623898 (291h45m15.797094359s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246035040 (291h45m15.797505491s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246184175 (291h45m15.797654636s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246797051 (291h45m15.798267522s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247334054 (291h45m15.798804515s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247548422 (291h45m15.799018873s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249260893 (291h45m15.800731354s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249559843 (291h45m15.801030304s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.251617881 (291h45m15.803088333s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.252868232 (291h45m15.804338704s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.254615760 (291h45m15.806086221s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:18.965747830 (291h45m17.517218281s) [upload=256935]: Next returned nil segment Nov 21 11:49:18.965769751 (291h45m17.517240243s) [upload=256935]: Waiting for error group managing segments... Nov 21 11:49:19.250399321 (291h45m17.801869792s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.250854327 (291h45m17.802324788s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.260959880 (291h45m17.812430341s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.262082247 (291h45m17.813552709s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.268139792 (291h45m17.819610263s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.280973680 (291h45m17.832444141s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.282868778 (291h45m17.83433924s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.286078351 (291h45m17.837548812s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.821632116 (291h45m18.373102577s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.824423853 (291h45m18.375894304s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.850707311 (291h45m18.402177772s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.908423644 (291h45m18.459894105s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958255192 (291h45m18.509725653s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958317902 (291h45m18.509788353s) [seg_pos={0 0}, upload=256935]: Segment reached optimal threshold of 13 pieces. Nov 21 11:49:19.958597444 (291h45m18.510067905s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection Nov 21 11:49:19.958607714 (291h45m18.510078175s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958655214 (291h45m18.510125675s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection Nov 21 11:49:19.958672887 (291h45m18.510143349s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958915429 (291h45m18.5103859s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Nov 21 11:49:19.958928194 (291h45m18.510398655s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958967489 (291h45m18.51043795s) [seg_pos={0 0}, upload=256935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:49:19.959064374 (291h45m18.510534835s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:49:19.959071227 (291h45m18.510541688s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:49:20.260011595 (291h45m18.811482076s) [upload=256935]: Done uploading segments... Nov 21 11:49:20.260040470 (291h45m18.811510931s) [upload=256935]: Upload finished. err: <nil> Duration: 6.375106081s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................] duration: 3.347371712s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:50:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.347397792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X...X.X......X.XX...X.X.X....X............X....X.X.X..X.X..X] duration: 30.003141561s trace: https://jaeger.dev.storj.io/trace/34d009b2527f9379 FAIL (2025-11-21 11:49:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ) context deadline exceeded ======LOGS====== ments... Nov 21 11:49:16.994612805 (291h45m15.546083266s) [upload=256935]: Got next segment. Inline: false Nov 21 11:49:16.994647011 (291h45m15.546117472s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:49:16.994651759 (291h45m15.54612221s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:49:17.243148153 (291h45m15.794618604s) [seg_pos={0 0}, upload=256935]: Begin upload segment... Nov 21 11:49:17.243197467 (291h45m15.794667929s) [seg_pos={0 0}, upload=256935]: Done begin upload segment. Nov 21 11:49:17.243223908 (291h45m15.794694369s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243288660 (291h45m15.794759112s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.243999364 (291h45m15.795469815s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.244784930 (291h45m15.796255391s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245522555 (291h45m15.796993016s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.245623898 (291h45m15.797094359s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246035040 (291h45m15.797505491s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246184175 (291h45m15.797654636s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.246797051 (291h45m15.798267522s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247334054 (291h45m15.798804515s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.247548422 (291h45m15.799018873s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249260893 (291h45m15.800731354s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.249559843 (291h45m15.801030304s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.251617881 (291h45m15.803088333s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.252868232 (291h45m15.804338704s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:17.254615760 (291h45m15.806086221s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Uploading piece... Nov 21 11:49:18.965747830 (291h45m17.517218281s) [upload=256935]: Next returned nil segment Nov 21 11:49:18.965769751 (291h45m17.517240243s) [upload=256935]: Waiting for error group managing segments... Nov 21 11:49:19.250399321 (291h45m17.801869792s) [noise=true , address=145.79.227.30:4761 , piece_id=SDNZAHFY6GR7I4L6BTFGXSAKA5O5MHJ54NHMA77VTAXTCWQQSFQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.250854327 (291h45m17.802324788s) [noise=true , address=145.79.227.10:4403 , piece_id=XS5LV3OLSHZWHVBGVPFV4TFGBPZGHMLGNF6OGVWWQQMTXVWC5QCQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.260959880 (291h45m17.812430341s) [noise=true , address=145.79.227.11:4731 , piece_id=2NO5DRKFPM22XIWMQU4GDYHKXWYQX2QRAYP3YQBSIWTLAMW6LAPA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.262082247 (291h45m17.813552709s) [noise=true , address=145.79.227.8:4330 , piece_id=ZCJFYOK2Q2B6DDNNWGW6TXFPUJMPOGLV6HAPQO2HEVCGXAQVGBRA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.268139792 (291h45m17.819610263s) [noise=true , address=145.79.227.28:4726 , piece_id=7XZ277AZOYL6KFSIAAHNTWNZ2PBWMT2YIQFRWNTYEIIINQ6VTRUQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.280973680 (291h45m17.832444141s) [noise=true , address=145.79.227.31:4798 , piece_id=VKGEFEBHEEU5VUVIVEV7OVY3SUPUXFVGNP7OWNJSBOQAKDYQFQGQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.282868778 (291h45m17.83433924s) [noise=true , address=145.79.227.23:4546 , piece_id=HIT5PPIDXBOIXNY6X3UPCJULOM7YX7BI4VST4IEP6VO7NPK7YMDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.286078351 (291h45m17.837548812s) [noise=true , address=145.79.227.8:4331 , piece_id=GK4K577UUQ7IM2NNN5Q4BCJ26JK3EET6VNEEWOPQO7I5VCLOGNFQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.821632116 (291h45m18.373102577s) [noise=true , address=77.90.31.13:4628 , piece_id=T3KCZD5JV6FAEG6VHBK6HG2E2C6XMP4A6NVAWFWKZTWJIA57L4AA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.824423853 (291h45m18.375894304s) [noise=true , address=77.90.31.14:4663 , piece_id=CECM54T2XTGYW4BFN4TZY2DYUH47J6YO6PGF2QQ6SOCUMLSW6XMQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.850707311 (291h45m18.402177772s) [noise=true , address=45.91.116.8:4048 , piece_id=N7NO3DOXYOQDEAGQ43OPWR3E5CBYZX5J4A4EJV2VLGKOKJBXCIQQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.908423644 (291h45m18.459894105s) [noise=true , address=77.90.31.11:4555 , piece_id=HLV5EPKQNPRWQKHNJLB5ATKRWSS6XW3HMWYXUJL4L7VM7VL4MBZQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958255192 (291h45m18.509725653s) [noise=true , address=45.91.116.14:4191 , piece_id=3TMBBNLJQ4OZ7RVYS7TYUILMTF5R7TQXXH5RRHDLFQMQAKM66WLA, seg_pos={0 0}, upload=256935]: Done uploading piece. err: <nil> Nov 21 11:49:19.958317902 (291h45m18.509788353s) [seg_pos={0 0}, upload=256935]: Segment reached optimal threshold of 13 pieces. Nov 21 11:49:19.958597444 (291h45m18.510067905s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49482->77.90.31.15:4698: use of closed network connection Nov 21 11:49:19.958607714 (291h45m18.510078175s) [noise=true , address=77.90.31.15:4698 , piece_id=QOPKTWSMOAHGWQ3XVTFJIBMWTJTZ3PJLI6U7BF4TDWR3ZTM2RG5Q, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958655214 (291h45m18.510125675s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57522->45.91.116.10:4081: use of closed network connection Nov 21 11:49:19.958672887 (291h45m18.510143349s) [noise=true , address=45.91.116.10:4081 , piece_id=CSMEKQE6SUSZ52HLGWVDXINNPRIYR6LJM5FBOI5RNFVH7VF7OLBQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958915429 (291h45m18.5103859s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Nov 21 11:49:19.958928194 (291h45m18.510398655s) [noise=true , address=45.91.116.4:4937 , piece_id=OKQ3DD3KZ73K3RRAE4RR6EG6HOYWZYGGJPKN2QTVN3BWWNKK7PDQ, seg_pos={0 0}, upload=256935]: Long tail cancellation due to optimal threshold Nov 21 11:49:19.958967489 (291h45m18.51043795s) [seg_pos={0 0}, upload=256935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:49:19.959064374 (291h45m18.510534835s) [upload=256935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:49:19.959071227 (291h45m18.510541688s) [upload=256935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:49:20.260011595 (291h45m18.811482076s) [upload=256935]: Done uploading segments... Nov 21 11:49:20.260040470 (291h45m18.811510931s) [upload=256935]: Upload finished. err: <nil> Duration: 30.003166049s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 5.53975258s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:49:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.539782757s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)