Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 906886 reports (Page 11247 of 18138)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X........X........] duration: 12.171232385s trace: https://jaeger.dev.storj.io/trace/5287a23df1c13acc FAIL (2025-12-30 12:54:15 UTC) orbiter warning: context deadline exceeded Duration: 12.171246361s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.......................X....X.] duration: 14.973504446s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 12:54:10 UTC) orbiter warning: context deadline exceeded Duration: 14.973529964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X........X.......] duration: 19.889142529s trace: https://jaeger.dev.storj.io/trace/5287a23df1c13acc FAIL (2025-12-30 12:54:03 UTC) orbiter warning: context deadline exceeded Duration: 19.889177016s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [...X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.] duration: 3.569550157s trace: https://jaeger.dev.storj.io/trace/4bcd451cebc1fdbf FAIL (2025-12-30 12:53:55 UTC) orbiter warning: context deadline exceeded Duration: 3.569574974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......................X....X] duration: 1m0.001275701s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 12:53:46 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001298634s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [....X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX] duration: 5.000523559s trace: https://jaeger.dev.storj.io/trace/4bcd451cebc1fdbf FAIL (2025-12-30 12:53:41 UTC) orbiter warning: context deadline exceeded Duration: 5.00054534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.XX....XX..X.......X.XX..X..X.XX...............X..X.X..X..] duration: 5.765320729s trace: https://jaeger.dev.storj.io/trace/7e4805d173e0a369 FAIL (2025-12-30 12:53:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA) context deadline exceeded ======LOGS====== Dec 30 12:52:14.115932176 (597h23m13.987015211s) [upload=523651]: Starting upload Dec 30 12:52:14.115994284 (597h23m13.987077329s) [upload=523651]: Done starting upload Dec 30 12:52:14.116134391 (597h23m13.987217436s) [upload=523651]: Uploading segments... Dec 30 12:52:14.116323982 (597h23m13.987407027s) [upload=523651]: Got next segment. Inline: false Dec 30 12:52:14.116390138 (597h23m13.987473173s) [upload=523651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:52:14.116396260 (597h23m13.987479315s) [upload=523651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:52:14.358767612 (597h23m14.229850647s) [seg_pos={0 0}, upload=523651]: Begin upload segment... Dec 30 12:52:14.358813980 (597h23m14.229897015s) [seg_pos={0 0}, upload=523651]: Done begin upload segment. Dec 30 12:52:14.358844779 (597h23m14.229927814s) [noise=true , address=145.79.227.10:4401 , piece_id=M56P2SXBJSURDWMWYZJLHZWQZEIEMNUURQ466SZANLUTHX4REE5Q, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.358995876 (597h23m14.230078911s) [noise=true , address=145.79.227.41:4087 , piece_id=YRQHXM2SXDBB55WWMHJRU4WNMCSNWM3ECQKLTCUK6LDIQUZHJYRQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.359937319 (597h23m14.231020364s) [noise=true , address=77.90.31.10:4518 , piece_id=KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.360144694 (597h23m14.231227739s) [noise=true , address=145.79.227.21:4473 , piece_id=H2BVWUFZGS5UY3CYE5EWKYMXX6LX3UZ735N3ER3EQNA7WTUYLLEA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.360694351 (597h23m14.231777396s) [noise=true , address=77.90.31.14:4661 , piece_id=CSWEMPTKJVW4GZC5DLDZUY5J43UALWK2UT7CY7MJTDKYUAJXCL7A, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.361171960 (597h23m14.232254995s) [noise=true , address=145.79.227.22:4510 , piece_id=QWT3E76ALSUOQITBKE4O2E5DNZE52IVGUECTIVOV44I7MNXXY6NA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.361831737 (597h23m14.232914782s) [noise=true , address=145.79.227.13:4804 , piece_id=4YDWGK35ZFVWPKGQDT4A5EBT23S5CBU6QANWEXZFDQK54C4S4QUQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.362325757 (597h23m14.233408802s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.362909749 (597h23m14.233992794s) [noise=true , address=145.79.227.28:4726 , piece_id=4B2OFKLPN3YIH46YSYNXXE6NVUM2AB7KEYQOAZYQMS2Z4SKTDUYA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.363590836 (597h23m14.234673881s) [noise=true , address=145.79.227.14:4838 , piece_id=WH2PMMJVZMBXKHUX3X3RGVD4IDVCZ4EYY3JBVOV7F2YMAHPFI2DA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.366121074 (597h23m14.237204149s) [noise=true , address=45.91.116.9:4263 , piece_id=HEOOYEEFNXNRGOQ45S6DPNNT5YZIZWIUG2VH4VKWLD7K62NP4ZQQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.366652215 (597h23m14.23773526s) [noise=true , address=145.79.227.15:4874 , piece_id=3R5H6XZYTULAWDWHTAN65I7PPPHOFXD5RNM35ZVUVYNH545YTXGQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.367079769 (597h23m14.238162804s) [noise=true , address=145.79.227.19:4078 , piece_id=MQIQVZ4KSF3ZQZEBS3ISSC2C4ETEBE2ZLQYQA3G2DOFD5C6FEUJA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.367563110 (597h23m14.238646145s) [noise=true , address=45.91.116.10:4083 , piece_id=IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.368083380 (597h23m14.239166415s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.370335168 (597h23m14.241418213s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:15.971132591 (597h23m15.842215636s) [upload=523651]: Next returned nil segment Dec 30 12:52:15.971151498 (597h23m15.842234533s) [upload=523651]: Waiting for error group managing segments... Dec 30 12:52:16.352891969 (597h23m16.223975014s) [noise=true , address=145.79.227.22:4510 , piece_id=QWT3E76ALSUOQITBKE4O2E5DNZE52IVGUECTIVOV44I7MNXXY6NA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.353001558 (597h23m16.224084593s) [noise=true , address=145.79.227.13:4804 , piece_id=4YDWGK35ZFVWPKGQDT4A5EBT23S5CBU6QANWEXZFDQK54C4S4QUQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.360107449 (597h23m16.231190504s) [noise=true , address=145.79.227.19:4078 , piece_id=MQIQVZ4KSF3ZQZEBS3ISSC2C4ETEBE2ZLQYQA3G2DOFD5C6FEUJA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.361588549 (597h23m16.232671594s) [noise=true , address=145.79.227.10:4401 , piece_id=M56P2SXBJSURDWMWYZJLHZWQZEIEMNUURQ466SZANLUTHX4REE5Q, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.370586382 (597h23m16.241669427s) [noise=true , address=145.79.227.28:4726 , piece_id=4B2OFKLPN3YIH46YSYNXXE6NVUM2AB7KEYQOAZYQMS2Z4SKTDUYA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.379901019 (597h23m16.250984054s) [noise=true , address=145.79.227.21:4473 , piece_id=H2BVWUFZGS5UY3CYE5EWKYMXX6LX3UZ735N3ER3EQNA7WTUYLLEA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.381883654 (597h23m16.252966699s) [noise=true , address=145.79.227.14:4838 , piece_id=WH2PMMJVZMBXKHUX3X3RGVD4IDVCZ4EYY3JBVOV7F2YMAHPFI2DA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.382184477 (597h23m16.253267512s) [noise=true , address=145.79.227.41:4087 , piece_id=YRQHXM2SXDBB55WWMHJRU4WNMCSNWM3ECQKLTCUK6LDIQUZHJYRQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.386081047 (597h23m16.257164082s) [noise=true , address=145.79.227.15:4874 , piece_id=3R5H6XZYTULAWDWHTAN65I7PPPHOFXD5RNM35ZVUVYNH545YTXGQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.648851922 (597h23m16.519934967s) [noise=true , address=77.90.31.14:4661 , piece_id=CSWEMPTKJVW4GZC5DLDZUY5J43UALWK2UT7CY7MJTDKYUAJXCL7A, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.690922852 (597h23m16.562005897s) [noise=true , address=77.90.31.10:4518 , piece_id=KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.814557211 (597h23m16.685640246s) [noise=true , address=45.91.116.9:4263 , piece_id=HEOOYEEFNXNRGOQ45S6DPNNT5YZIZWIUG2VH4VKWLD7K62NP4ZQQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:17.475466253 (597h23m17.346549298s) [noise=true , address=45.91.116.10:4083 , piece_id=IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:17.475491310 (597h23m17.346574355s) [seg_pos={0 0}, upload=523651]: Segment reached optimal threshold of 13 pieces. Dec 30 12:52:17.475700448 (597h23m17.346783493s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 12:52:17.475708314 (597h23m17.346791359s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold Dec 30 12:52:17.476027872 (597h23m17.347110907s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 12:52:17.476036999 (597h23m17.347120034s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold Dec 30 12:52:17.476181614 (597h23m17.347264659s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 30 12:52:17.476189659 (597h23m17.347272705s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold Dec 30 12:52:17.476211431 (597h23m17.347294476s) [seg_pos={0 0}, upload=523651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:52:17.476287837 (597h23m17.347370882s) [upload=523651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:52:17.476293798 (597h23m17.347376843s) [upload=523651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:52:17.763390204 (597h23m17.634473239s) [upload=523651]: Done uploading segments... Dec 30 12:52:17.763411815 (597h23m17.63449485s) [upload=523651]: Upload finished. err: <nil> Duration: 5.765347109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X..................X.....X.X...X.........X....X........X..] duration: 15.280224968s trace: https://jaeger.dev.storj.io/trace/1e6433ec4a141391 FAIL (2025-12-30 12:53:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments... Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces. Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments... Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil> Duration: 15.280242782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X........X......] duration: 13.968859397s trace: https://jaeger.dev.storj.io/trace/5287a23df1c13acc FAIL (2025-12-30 12:53:22 UTC) orbiter warning: context deadline exceeded Duration: 13.968874637s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX.XX....XX..X.......X.XX..X..X.XX...............X..X.X..X.] duration: 5.778970878s trace: https://jaeger.dev.storj.io/trace/7e4805d173e0a369 FAIL (2025-12-30 12:52:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA) context deadline exceeded ======LOGS====== Dec 30 12:52:14.115932176 (597h23m13.987015211s) [upload=523651]: Starting upload Dec 30 12:52:14.115994284 (597h23m13.987077329s) [upload=523651]: Done starting upload Dec 30 12:52:14.116134391 (597h23m13.987217436s) [upload=523651]: Uploading segments... Dec 30 12:52:14.116323982 (597h23m13.987407027s) [upload=523651]: Got next segment. Inline: false Dec 30 12:52:14.116390138 (597h23m13.987473173s) [upload=523651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:52:14.116396260 (597h23m13.987479315s) [upload=523651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:52:14.358767612 (597h23m14.229850647s) [seg_pos={0 0}, upload=523651]: Begin upload segment... Dec 30 12:52:14.358813980 (597h23m14.229897015s) [seg_pos={0 0}, upload=523651]: Done begin upload segment. Dec 30 12:52:14.358844779 (597h23m14.229927814s) [noise=true , address=145.79.227.10:4401 , piece_id=M56P2SXBJSURDWMWYZJLHZWQZEIEMNUURQ466SZANLUTHX4REE5Q, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.358995876 (597h23m14.230078911s) [noise=true , address=145.79.227.41:4087 , piece_id=YRQHXM2SXDBB55WWMHJRU4WNMCSNWM3ECQKLTCUK6LDIQUZHJYRQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.359937319 (597h23m14.231020364s) [noise=true , address=77.90.31.10:4518 , piece_id=KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.360144694 (597h23m14.231227739s) [noise=true , address=145.79.227.21:4473 , piece_id=H2BVWUFZGS5UY3CYE5EWKYMXX6LX3UZ735N3ER3EQNA7WTUYLLEA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.360694351 (597h23m14.231777396s) [noise=true , address=77.90.31.14:4661 , piece_id=CSWEMPTKJVW4GZC5DLDZUY5J43UALWK2UT7CY7MJTDKYUAJXCL7A, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.361171960 (597h23m14.232254995s) [noise=true , address=145.79.227.22:4510 , piece_id=QWT3E76ALSUOQITBKE4O2E5DNZE52IVGUECTIVOV44I7MNXXY6NA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.361831737 (597h23m14.232914782s) [noise=true , address=145.79.227.13:4804 , piece_id=4YDWGK35ZFVWPKGQDT4A5EBT23S5CBU6QANWEXZFDQK54C4S4QUQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.362325757 (597h23m14.233408802s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.362909749 (597h23m14.233992794s) [noise=true , address=145.79.227.28:4726 , piece_id=4B2OFKLPN3YIH46YSYNXXE6NVUM2AB7KEYQOAZYQMS2Z4SKTDUYA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.363590836 (597h23m14.234673881s) [noise=true , address=145.79.227.14:4838 , piece_id=WH2PMMJVZMBXKHUX3X3RGVD4IDVCZ4EYY3JBVOV7F2YMAHPFI2DA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.366121074 (597h23m14.237204149s) [noise=true , address=45.91.116.9:4263 , piece_id=HEOOYEEFNXNRGOQ45S6DPNNT5YZIZWIUG2VH4VKWLD7K62NP4ZQQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.366652215 (597h23m14.23773526s) [noise=true , address=145.79.227.15:4874 , piece_id=3R5H6XZYTULAWDWHTAN65I7PPPHOFXD5RNM35ZVUVYNH545YTXGQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.367079769 (597h23m14.238162804s) [noise=true , address=145.79.227.19:4078 , piece_id=MQIQVZ4KSF3ZQZEBS3ISSC2C4ETEBE2ZLQYQA3G2DOFD5C6FEUJA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.367563110 (597h23m14.238646145s) [noise=true , address=45.91.116.10:4083 , piece_id=IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.368083380 (597h23m14.239166415s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.370335168 (597h23m14.241418213s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:15.971132591 (597h23m15.842215636s) [upload=523651]: Next returned nil segment Dec 30 12:52:15.971151498 (597h23m15.842234533s) [upload=523651]: Waiting for error group managing segments... Dec 30 12:52:16.352891969 (597h23m16.223975014s) [noise=true , address=145.79.227.22:4510 , piece_id=QWT3E76ALSUOQITBKE4O2E5DNZE52IVGUECTIVOV44I7MNXXY6NA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.353001558 (597h23m16.224084593s) [noise=true , address=145.79.227.13:4804 , piece_id=4YDWGK35ZFVWPKGQDT4A5EBT23S5CBU6QANWEXZFDQK54C4S4QUQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.360107449 (597h23m16.231190504s) [noise=true , address=145.79.227.19:4078 , piece_id=MQIQVZ4KSF3ZQZEBS3ISSC2C4ETEBE2ZLQYQA3G2DOFD5C6FEUJA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.361588549 (597h23m16.232671594s) [noise=true , address=145.79.227.10:4401 , piece_id=M56P2SXBJSURDWMWYZJLHZWQZEIEMNUURQ466SZANLUTHX4REE5Q, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.370586382 (597h23m16.241669427s) [noise=true , address=145.79.227.28:4726 , piece_id=4B2OFKLPN3YIH46YSYNXXE6NVUM2AB7KEYQOAZYQMS2Z4SKTDUYA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.379901019 (597h23m16.250984054s) [noise=true , address=145.79.227.21:4473 , piece_id=H2BVWUFZGS5UY3CYE5EWKYMXX6LX3UZ735N3ER3EQNA7WTUYLLEA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.381883654 (597h23m16.252966699s) [noise=true , address=145.79.227.14:4838 , piece_id=WH2PMMJVZMBXKHUX3X3RGVD4IDVCZ4EYY3JBVOV7F2YMAHPFI2DA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.382184477 (597h23m16.253267512s) [noise=true , address=145.79.227.41:4087 , piece_id=YRQHXM2SXDBB55WWMHJRU4WNMCSNWM3ECQKLTCUK6LDIQUZHJYRQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.386081047 (597h23m16.257164082s) [noise=true , address=145.79.227.15:4874 , piece_id=3R5H6XZYTULAWDWHTAN65I7PPPHOFXD5RNM35ZVUVYNH545YTXGQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.648851922 (597h23m16.519934967s) [noise=true , address=77.90.31.14:4661 , piece_id=CSWEMPTKJVW4GZC5DLDZUY5J43UALWK2UT7CY7MJTDKYUAJXCL7A, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.690922852 (597h23m16.562005897s) [noise=true , address=77.90.31.10:4518 , piece_id=KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.814557211 (597h23m16.685640246s) [noise=true , address=45.91.116.9:4263 , piece_id=HEOOYEEFNXNRGOQ45S6DPNNT5YZIZWIUG2VH4VKWLD7K62NP4ZQQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:17.475466253 (597h23m17.346549298s) [noise=true , address=45.91.116.10:4083 , piece_id=IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:17.475491310 (597h23m17.346574355s) [seg_pos={0 0}, upload=523651]: Segment reached optimal threshold of 13 pieces. Dec 30 12:52:17.475700448 (597h23m17.346783493s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 12:52:17.475708314 (597h23m17.346791359s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold Dec 30 12:52:17.476027872 (597h23m17.347110907s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 12:52:17.476036999 (597h23m17.347120034s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold Dec 30 12:52:17.476181614 (597h23m17.347264659s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 30 12:52:17.476189659 (597h23m17.347272705s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold Dec 30 12:52:17.476211431 (597h23m17.347294476s) [seg_pos={0 0}, upload=523651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:52:17.476287837 (597h23m17.347370882s) [upload=523651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:52:17.476293798 (597h23m17.347376843s) [upload=523651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:52:17.763390204 (597h23m17.634473239s) [upload=523651]: Done uploading segments... Dec 30 12:52:17.763411815 (597h23m17.63449485s) [upload=523651]: Upload finished. err: <nil> Duration: 5.778999513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X..................X.....X.X...X.........X....X........X.] duration: 15.590850374s trace: https://jaeger.dev.storj.io/trace/1e6433ec4a141391 FAIL (2025-12-30 12:52:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments... Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces. Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments... Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil> Duration: 15.590868607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX.XX....XX..X.......X.XX..X..X.XX...............X..X.X..X] duration: 30.002961296s trace: https://jaeger.dev.storj.io/trace/7e4805d173e0a369 FAIL (2025-12-30 12:52:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA) context deadline exceeded ======LOGS====== Dec 30 12:52:14.115932176 (597h23m13.987015211s) [upload=523651]: Starting upload Dec 30 12:52:14.115994284 (597h23m13.987077329s) [upload=523651]: Done starting upload Dec 30 12:52:14.116134391 (597h23m13.987217436s) [upload=523651]: Uploading segments... Dec 30 12:52:14.116323982 (597h23m13.987407027s) [upload=523651]: Got next segment. Inline: false Dec 30 12:52:14.116390138 (597h23m13.987473173s) [upload=523651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:52:14.116396260 (597h23m13.987479315s) [upload=523651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:52:14.358767612 (597h23m14.229850647s) [seg_pos={0 0}, upload=523651]: Begin upload segment... Dec 30 12:52:14.358813980 (597h23m14.229897015s) [seg_pos={0 0}, upload=523651]: Done begin upload segment. Dec 30 12:52:14.358844779 (597h23m14.229927814s) [noise=true , address=145.79.227.10:4401 , piece_id=M56P2SXBJSURDWMWYZJLHZWQZEIEMNUURQ466SZANLUTHX4REE5Q, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.358995876 (597h23m14.230078911s) [noise=true , address=145.79.227.41:4087 , piece_id=YRQHXM2SXDBB55WWMHJRU4WNMCSNWM3ECQKLTCUK6LDIQUZHJYRQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.359937319 (597h23m14.231020364s) [noise=true , address=77.90.31.10:4518 , piece_id=KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.360144694 (597h23m14.231227739s) [noise=true , address=145.79.227.21:4473 , piece_id=H2BVWUFZGS5UY3CYE5EWKYMXX6LX3UZ735N3ER3EQNA7WTUYLLEA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.360694351 (597h23m14.231777396s) [noise=true , address=77.90.31.14:4661 , piece_id=CSWEMPTKJVW4GZC5DLDZUY5J43UALWK2UT7CY7MJTDKYUAJXCL7A, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.361171960 (597h23m14.232254995s) [noise=true , address=145.79.227.22:4510 , piece_id=QWT3E76ALSUOQITBKE4O2E5DNZE52IVGUECTIVOV44I7MNXXY6NA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.361831737 (597h23m14.232914782s) [noise=true , address=145.79.227.13:4804 , piece_id=4YDWGK35ZFVWPKGQDT4A5EBT23S5CBU6QANWEXZFDQK54C4S4QUQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.362325757 (597h23m14.233408802s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.362909749 (597h23m14.233992794s) [noise=true , address=145.79.227.28:4726 , piece_id=4B2OFKLPN3YIH46YSYNXXE6NVUM2AB7KEYQOAZYQMS2Z4SKTDUYA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.363590836 (597h23m14.234673881s) [noise=true , address=145.79.227.14:4838 , piece_id=WH2PMMJVZMBXKHUX3X3RGVD4IDVCZ4EYY3JBVOV7F2YMAHPFI2DA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.366121074 (597h23m14.237204149s) [noise=true , address=45.91.116.9:4263 , piece_id=HEOOYEEFNXNRGOQ45S6DPNNT5YZIZWIUG2VH4VKWLD7K62NP4ZQQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.366652215 (597h23m14.23773526s) [noise=true , address=145.79.227.15:4874 , piece_id=3R5H6XZYTULAWDWHTAN65I7PPPHOFXD5RNM35ZVUVYNH545YTXGQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.367079769 (597h23m14.238162804s) [noise=true , address=145.79.227.19:4078 , piece_id=MQIQVZ4KSF3ZQZEBS3ISSC2C4ETEBE2ZLQYQA3G2DOFD5C6FEUJA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.367563110 (597h23m14.238646145s) [noise=true , address=45.91.116.10:4083 , piece_id=IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.368083380 (597h23m14.239166415s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:14.370335168 (597h23m14.241418213s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Uploading piece... Dec 30 12:52:15.971132591 (597h23m15.842215636s) [upload=523651]: Next returned nil segment Dec 30 12:52:15.971151498 (597h23m15.842234533s) [upload=523651]: Waiting for error group managing segments... Dec 30 12:52:16.352891969 (597h23m16.223975014s) [noise=true , address=145.79.227.22:4510 , piece_id=QWT3E76ALSUOQITBKE4O2E5DNZE52IVGUECTIVOV44I7MNXXY6NA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.353001558 (597h23m16.224084593s) [noise=true , address=145.79.227.13:4804 , piece_id=4YDWGK35ZFVWPKGQDT4A5EBT23S5CBU6QANWEXZFDQK54C4S4QUQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.360107449 (597h23m16.231190504s) [noise=true , address=145.79.227.19:4078 , piece_id=MQIQVZ4KSF3ZQZEBS3ISSC2C4ETEBE2ZLQYQA3G2DOFD5C6FEUJA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.361588549 (597h23m16.232671594s) [noise=true , address=145.79.227.10:4401 , piece_id=M56P2SXBJSURDWMWYZJLHZWQZEIEMNUURQ466SZANLUTHX4REE5Q, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.370586382 (597h23m16.241669427s) [noise=true , address=145.79.227.28:4726 , piece_id=4B2OFKLPN3YIH46YSYNXXE6NVUM2AB7KEYQOAZYQMS2Z4SKTDUYA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.379901019 (597h23m16.250984054s) [noise=true , address=145.79.227.21:4473 , piece_id=H2BVWUFZGS5UY3CYE5EWKYMXX6LX3UZ735N3ER3EQNA7WTUYLLEA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.381883654 (597h23m16.252966699s) [noise=true , address=145.79.227.14:4838 , piece_id=WH2PMMJVZMBXKHUX3X3RGVD4IDVCZ4EYY3JBVOV7F2YMAHPFI2DA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.382184477 (597h23m16.253267512s) [noise=true , address=145.79.227.41:4087 , piece_id=YRQHXM2SXDBB55WWMHJRU4WNMCSNWM3ECQKLTCUK6LDIQUZHJYRQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.386081047 (597h23m16.257164082s) [noise=true , address=145.79.227.15:4874 , piece_id=3R5H6XZYTULAWDWHTAN65I7PPPHOFXD5RNM35ZVUVYNH545YTXGQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.648851922 (597h23m16.519934967s) [noise=true , address=77.90.31.14:4661 , piece_id=CSWEMPTKJVW4GZC5DLDZUY5J43UALWK2UT7CY7MJTDKYUAJXCL7A, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.690922852 (597h23m16.562005897s) [noise=true , address=77.90.31.10:4518 , piece_id=KUNKLOKIRSYDCFDOMUXCUUBGR4CZTZUKSRLKLZI5NLBD5QMNGTCQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:16.814557211 (597h23m16.685640246s) [noise=true , address=45.91.116.9:4263 , piece_id=HEOOYEEFNXNRGOQ45S6DPNNT5YZIZWIUG2VH4VKWLD7K62NP4ZQQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:17.475466253 (597h23m17.346549298s) [noise=true , address=45.91.116.10:4083 , piece_id=IZTI4X4FSVD3K563B5VKKZDAHMUBW2ZDQHZATZNOZMSKKZM7FXGA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: <nil> Dec 30 12:52:17.475491310 (597h23m17.346574355s) [seg_pos={0 0}, upload=523651]: Segment reached optimal threshold of 13 pieces. Dec 30 12:52:17.475700448 (597h23m17.346783493s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 12:52:17.475708314 (597h23m17.346791359s) [noise=true , address=145.79.227.34:4905 , piece_id=JEEM7QYBUR7X7675WIJJG4PTLSZR53BL45HV6MZKTDD65KCFDLRQ, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold Dec 30 12:52:17.476027872 (597h23m17.347110907s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 12:52:17.476036999 (597h23m17.347120034s) [noise=true , address=145.79.227.29:4943 , piece_id=JPE7GTBBRY54NABCPQJFJJMN3YEVY3RLODE6JIFTEMMNKUL3RMIA, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold Dec 30 12:52:17.476181614 (597h23m17.347264659s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 30 12:52:17.476189659 (597h23m17.347272705s) [noise=true , address=145.79.227.30:4761 , piece_id=EVMZUUTYJKJRDHAONOCRJRZQL54FIMWWKTFPS5KVYUZAOCHL4XOQ, seg_pos={0 0}, upload=523651]: Long tail cancellation due to optimal threshold Dec 30 12:52:17.476211431 (597h23m17.347294476s) [seg_pos={0 0}, upload=523651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:52:17.476287837 (597h23m17.347370882s) [upload=523651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:52:17.476293798 (597h23m17.347376843s) [upload=523651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:52:17.763390204 (597h23m17.634473239s) [upload=523651]: Done uploading segments... Dec 30 12:52:17.763411815 (597h23m17.63449485s) [upload=523651]: Upload finished. err: <nil> Duration: 30.002986263s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [.....X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXX] duration: 5.001915478s trace: https://jaeger.dev.storj.io/trace/6838d809a25af3ea FAIL (2025-12-30 12:52:40 UTC) orbiter warning: context deadline exceeded Duration: 5.001932681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..................X.....X.X...X.........X....X........X] duration: 1m0.005800461s trace: https://jaeger.dev.storj.io/trace/1e6433ec4a141391 FAIL (2025-12-30 12:52:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.352305080 (597h22m35.223388115s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353083031 (597h22m35.224166066s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.353482140 (597h22m35.224565175s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354315267 (597h22m35.225398302s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.354723675 (597h22m35.22580671s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.355352552 (597h22m35.226435587s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356361994 (597h22m35.227445029s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.356578216 (597h22m35.227661261s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.357840370 (597h22m35.228923405s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359039433 (597h22m35.230122468s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.359333493 (597h22m35.230416538s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.360694154 (597h22m35.231777189s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.361764952 (597h22m35.232847987s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363365710 (597h22m35.234448755s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:35.363684648 (597h22m35.234767683s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Uploading piece... Dec 30 12:51:36.749009517 (597h22m36.620092552s) [upload=523642]: Next returned nil segment Dec 30 12:51:36.749023393 (597h22m36.620106428s) [upload=523642]: Waiting for error group managing segments... Dec 30 12:51:37.208596330 (597h22m37.079679376s) [noise=true , address=145.79.227.17:4991 , piece_id=GUPSOAIXZGPRJ7MVZ46464IRG7FBQEFR2RHYDT2GIPAMAKN36YHQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.240881687 (597h22m37.111964722s) [noise=true , address=145.79.227.49:4367 , piece_id=6L53GGMES5PBPHPLUCPDUGYKKOQQ64VSZWAGX6EP755TJXT5RH6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.294293140 (597h22m37.165376175s) [noise=true , address=145.79.227.2:4119 , piece_id=55AC2GONNJPSGZJ3Y3CYFRLUVKRDMZGLY4HMAPDMEGMN6AJCYWOQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.302040142 (597h22m37.173123177s) [noise=true , address=145.79.227.43:4160 , piece_id=746FBAMI37I2B2UMEKDGOVGNDHN5EXJJH6CMGSCTEEOAVLNDXVXA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.308919160 (597h22m37.180002196s) [noise=true , address=145.79.227.19:4099 , piece_id=DLLN7EUVR2D72XD7BL57AW5JPHVE76PUWF6LWXDEZQH6JZVH3RZQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.370286944 (597h22m37.24136999s) [noise=true , address=77.90.31.10:4545 , piece_id=YQV3IPXMIBSO5ZCISFUBGRGHE6FIXLKTLNYKPWR42VNBKNOGPG5Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.397564325 (597h22m37.26864737s) [noise=true , address=77.90.31.18:4812 , piece_id=V347IHYDQTVIQ43JTHQJJW2BU3E4VSOSWABCPJPVX2JVCR6J6UNA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.481372759 (597h22m37.352455794s) [noise=true , address=77.90.31.14:4682 , piece_id=XACXXUOJEEGBXQEJFZYOZKQDX5N2V3LX5NCEQP64YIBZPHK23NQA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:37.868609147 (597h22m37.739692192s) [noise=true , address=145.79.227.46:4612 , piece_id=VG2ZGVJNHBN2GT72UXK6QA32HYGW6465MEOII6GYSOS7SZYLBBJA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.122711215 (597h22m37.993794251s) [noise=true , address=145.79.227.44:4222 , piece_id=32QNUAYLXL4AQYBRI6LDICOWQIZMSU3OVN5SWVOCB6XIZY4FHRSA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.156332275 (597h22m38.02741531s) [noise=true , address=145.79.227.30:4775 , piece_id=JADWMGGTCCSYQMINZV4CLHKEAWREFOS5S4RSY2MFBKC3ULFBD7FA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.210830938 (597h22m38.081913983s) [noise=true , address=145.79.227.34:4925 , piece_id=G46JE4KCDC7FPDEUW2YDTSLI5X66MAMDCRRLDG7EBYF6ENLELLEQ, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.318916249 (597h22m38.189999284s) [noise=true , address=145.79.227.33:4897 , piece_id=6SGUOB6IMRF2SPO6WHSKB6XLQENVS5TCJ6RMO6BYIMQ2PANMHK6A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528107081 (597h22m38.399190127s) [noise=true , address=193.149.164.143:4694 , piece_id=AOZVLCCSBEYZX3WES7JJVOA2BECEBPII3A33PG5KCSQY6LSZ6Y3A, seg_pos={0 0}, upload=523642]: Done uploading piece. err: <nil> Dec 30 12:51:38.528133793 (597h22m38.399216838s) [seg_pos={0 0}, upload=523642]: Segment reached optimal threshold of 14 pieces. Dec 30 12:51:38.528364652 (597h22m38.399447687s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528373429 (597h22m38.399456464s) [noise=true , address=45.91.116.9:4289 , piece_id=O735F2PWWAXGOP54RXURLZ64S7XC4XZZY4OH7RM4N2UOIAA3LJTA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528672969 (597h22m38.399756014s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528682567 (597h22m38.399765602s) [noise=true , address=145.79.227.40:4072 , piece_id=ACK67AVKO5NK7IJKXGL2SLN4GGEKGW47WNTCZCPHIAKIVXYGFHVA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.528839085 (597h22m38.39992212s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ttdx6LHmuAt8JVC7arCDgWjm9eLadjeaEw17F12btoXuSiq9t): protocol: expected piece hash; context canceled; EOF Dec 30 12:51:38.528847421 (597h22m38.399930476s) [noise=true , address=77.90.31.11:4583 , piece_id=JEMHBN5CJQI4GTDPRB5RZLBRDUWB46JL7JTORRZXYJTLFK7Z66PA, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529001655 (597h22m38.4000847s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->64.49.13.169:4716: use of closed network connection Dec 30 12:51:38.529010362 (597h22m38.400093407s) [noise=true , address=64.49.13.169:4716 , piece_id=EIRJU5OSYOF3D3AWCGKKSPFL36THXWUWJWIMBYHVEPI7VB6ATD3Q, seg_pos={0 0}, upload=523642]: Long tail cancellation due to optimal threshold Dec 30 12:51:38.529029568 (597h22m38.400112603s) [seg_pos={0 0}, upload=523642]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:51:38.529107577 (597h22m38.400190612s) [upload=523642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:51:38.529114079 (597h22m38.400197124s) [upload=523642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:51:38.828356312 (597h22m38.699439387s) [upload=523642]: Done uploading segments... Dec 30 12:51:38.828379166 (597h22m38.699462231s) [upload=523642]: Upload finished. err: <nil> Duration: 1m0.005813045s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X........X.....] duration: 7.322364125s trace: https://jaeger.dev.storj.io/trace/5287a23df1c13acc FAIL (2025-12-30 12:52:34 UTC) orbiter warning: context deadline exceeded Duration: 7.322393191s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.......................X....] duration: 12.506246876s trace: https://jaeger.dev.storj.io/trace/2de4b260b3332c21 FAIL (2025-12-30 12:52:23 UTC) orbiter warning: context deadline exceeded Duration: 12.506274459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X........X....] duration: 16.205219591s trace: https://jaeger.dev.storj.io/trace/5287a23df1c13acc FAIL (2025-12-30 12:52:06 UTC) orbiter warning: context deadline exceeded Duration: 16.205249771s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X.......................X...] duration: 15.20165283s trace: https://jaeger.dev.storj.io/trace/2de4b260b3332c21 FAIL (2025-12-30 12:52:02 UTC) orbiter warning: context deadline exceeded Duration: 15.201684159s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [X.....X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXX] duration: 5.00092109s trace: https://jaeger.dev.storj.io/trace/583aa0c3e1bb8069 FAIL (2025-12-30 12:51:50 UTC) orbiter warning: context deadline exceeded Duration: 5.00094753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...XX.XX....XX..X.......X.XX..X..X.XX...............X..X.X..] duration: 6.189628808s trace: https://jaeger.dev.storj.io/trace/320d45110f7ef5b6 FAIL (2025-12-30 12:51:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 12:49:39.837093183 (597h20m39.708176228s) [upload=523610]: Starting upload Dec 30 12:49:39.837180680 (597h20m39.708263725s) [upload=523610]: Done starting upload Dec 30 12:49:39.837271182 (597h20m39.708354217s) [upload=523610]: Uploading segments... Dec 30 12:49:39.837525856 (597h20m39.708608891s) [upload=523610]: Got next segment. Inline: false Dec 30 12:49:39.837598575 (597h20m39.70868161s) [upload=523610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:49:39.837604036 (597h20m39.708687081s) [upload=523610]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:49:40.079647152 (597h20m39.950730197s) [seg_pos={0 0}, upload=523610]: Begin upload segment... Dec 30 12:49:40.079689804 (597h20m39.950772839s) [seg_pos={0 0}, upload=523610]: Done begin upload segment. Dec 30 12:49:40.079714861 (597h20m39.950797896s) [noise=true , address=145.79.227.26:4653 , piece_id=XHJSJEABGBMUCWGQBK34SHF2LQMN5SVTREFVJK5R4BHDWL4UZEUQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.079853104 (597h20m39.950936159s) [noise=true , address=145.79.227.10:4401 , piece_id=LUUDBIPEIPQZK43MXNKYHT2L5G2J7DS7IZ352IV4O2OQOJAMY2TA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.080372202 (597h20m39.951455237s) [noise=true , address=77.90.31.17:4773 , piece_id=37B2HCJC3LXKWYYYGXLGDX3WD2B3THUSW3I7IBF4RKBO6UBK7LHA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.080795949 (597h20m39.951878994s) [noise=true , address=145.79.227.36:4517 , piece_id=CLDJH2NRDZTLKLF4VB53OX4MTMBZ5HWCRJE2HVTYIKHV7XX5NJFA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.081173869 (597h20m39.952256904s) [noise=true , address=145.79.227.51:4410 , piece_id=C7M2U67WHWPJN7KPPZKHAY5O7SYP5A777WN7HVOQYGDIN2X6UQUQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.081568982 (597h20m39.952652027s) [noise=true , address=145.79.227.41:4085 , piece_id=XWVPP5HFUN6QEE4OOA5CN6L666DXEKOCNETLS3OEUHB5EGPI2DSQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.082005152 (597h20m39.953088187s) [noise=true , address=145.79.227.52:4447 , piece_id=RP6HLWFCJFTSAK2GTWNTWWFMBJJJ7INKTJDQOEISHTIGDCIXOMHQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.082407448 (597h20m39.953490483s) [noise=true , address=145.79.227.45:4231 , piece_id=565BVE3ALKOSWAOQCW52WUPWB5BSYVWMT5JZWTF4TBNTMCZTJ7UQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.083239231 (597h20m39.954322276s) [noise=true , address=45.91.116.14:4189 , piece_id=IXCXLOA3SN633DBOPI6NMX3YHZ4B5A5OZQJGVBX64TYAGAAV3ALQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.083670193 (597h20m39.954753228s) [noise=true , address=145.79.227.5:4221 , piece_id=RG6KPTD5J6SSLT6XKRLXK7NYPN3YHGGM4GW6BXTVQRPUWY2NARLA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.084370256 (597h20m39.955453291s) [noise=true , address=77.90.31.19:4841 , piece_id=WO27AQ7XCOG4KD22L2Y46C6HIANJ7IZHQ7X4JZWW45UCVHJA4NTQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.085458187 (597h20m39.956541222s) [noise=true , address=145.79.227.23:4547 , piece_id=OWINTYKOFJCUH5OL2PY3GO56WZS2IXFG6IH5WXXTPXYEPFQ23UXA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.086559975 (597h20m39.95764301s) [noise=true , address=45.91.116.4:4937 , piece_id=R2PNVVXRNFO6DVL64DCRHWJHV2CF7YFU744Q3QCQ3REVL7NFPV7Q, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.088229825 (597h20m39.95931286s) [noise=true , address=145.79.227.30:4761 , piece_id=RG6XRIBU2DD27WV7YGFKFYDGS33JJKFK6WFBTL6FLOFJMT3V5ELQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.089374163 (597h20m39.960457238s) [noise=true , address=145.79.227.3:4151 , piece_id=D3CMHDX3CJJVPQTD5I2LRX2TDAXE64AF5GCWIBPIH653HCBEHHAA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.090547417 (597h20m39.961630453s) [noise=true , address=45.91.116.5:4299 , piece_id=XDIIO3XXSWY5HZE72C5L4OZCK25LWMJCTCFGAUECYJOY5YXK23TQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:41.785690441 (597h20m41.656773486s) [upload=523610]: Next returned nil segment Dec 30 12:49:41.785704808 (597h20m41.656787844s) [upload=523610]: Waiting for error group managing segments... Dec 30 12:49:42.068560217 (597h20m41.939643262s) [noise=true , address=145.79.227.52:4447 , piece_id=RP6HLWFCJFTSAK2GTWNTWWFMBJJJ7INKTJDQOEISHTIGDCIXOMHQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.074125804 (597h20m41.94520884s) [noise=true , address=145.79.227.5:4221 , piece_id=RG6KPTD5J6SSLT6XKRLXK7NYPN3YHGGM4GW6BXTVQRPUWY2NARLA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.074506950 (597h20m41.945589995s) [noise=true , address=145.79.227.51:4410 , piece_id=C7M2U67WHWPJN7KPPZKHAY5O7SYP5A777WN7HVOQYGDIN2X6UQUQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.074770032 (597h20m41.945853067s) [noise=true , address=145.79.227.36:4517 , piece_id=CLDJH2NRDZTLKLF4VB53OX4MTMBZ5HWCRJE2HVTYIKHV7XX5NJFA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.076453878 (597h20m41.947536913s) [noise=true , address=145.79.227.23:4547 , piece_id=OWINTYKOFJCUH5OL2PY3GO56WZS2IXFG6IH5WXXTPXYEPFQ23UXA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.094770926 (597h20m41.965853961s) [noise=true , address=145.79.227.45:4231 , piece_id=565BVE3ALKOSWAOQCW52WUPWB5BSYVWMT5JZWTF4TBNTMCZTJ7UQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.095653527 (597h20m41.966736572s) [noise=true , address=145.79.227.41:4085 , piece_id=XWVPP5HFUN6QEE4OOA5CN6L666DXEKOCNETLS3OEUHB5EGPI2DSQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.098377954 (597h20m41.969460989s) [noise=true , address=145.79.227.3:4151 , piece_id=D3CMHDX3CJJVPQTD5I2LRX2TDAXE64AF5GCWIBPIH653HCBEHHAA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.150555228 (597h20m42.021638273s) [noise=true , address=145.79.227.10:4401 , piece_id=LUUDBIPEIPQZK43MXNKYHT2L5G2J7DS7IZ352IV4O2OQOJAMY2TA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.176734807 (597h20m42.047817842s) [noise=true , address=145.79.227.26:4653 , piece_id=XHJSJEABGBMUCWGQBK34SHF2LQMN5SVTREFVJK5R4BHDWL4UZEUQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.257699486 (597h20m42.128782541s) [noise=true , address=77.90.31.19:4841 , piece_id=WO27AQ7XCOG4KD22L2Y46C6HIANJ7IZHQ7X4JZWW45UCVHJA4NTQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.408523477 (597h20m42.279606512s) [noise=true , address=45.91.116.14:4189 , piece_id=IXCXLOA3SN633DBOPI6NMX3YHZ4B5A5OZQJGVBX64TYAGAAV3ALQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.415667241 (597h20m42.286750276s) [noise=true , address=45.91.116.5:4299 , piece_id=XDIIO3XXSWY5HZE72C5L4OZCK25LWMJCTCFGAUECYJOY5YXK23TQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.415691096 (597h20m42.286774131s) [seg_pos={0 0}, upload=523610]: Segment reached optimal threshold of 13 pieces. Dec 30 12:49:42.415868994 (597h20m42.286952039s) [noise=true , address=145.79.227.30:4761 , piece_id=RG6XRIBU2DD27WV7YGFKFYDGS33JJKFK6WFBTL6FLOFJMT3V5ELQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 30 12:49:42.415875267 (597h20m42.286958302s) [noise=true , address=145.79.227.30:4761 , piece_id=RG6XRIBU2DD27WV7YGFKFYDGS33JJKFK6WFBTL6FLOFJMT3V5ELQ, seg_pos={0 0}, upload=523610]: Long tail cancellation due to optimal threshold Dec 30 12:49:42.416000425 (597h20m42.28708346s) [noise=true , address=77.90.31.17:4773 , piece_id=37B2HCJC3LXKWYYYGXLGDX3WD2B3THUSW3I7IBF4RKBO6UBK7LHA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:43136->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:43136->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43136->77.90.31.17:4773: use of closed network connection Dec 30 12:49:42.416015775 (597h20m42.28709882s) [noise=true , address=77.90.31.17:4773 , piece_id=37B2HCJC3LXKWYYYGXLGDX3WD2B3THUSW3I7IBF4RKBO6UBK7LHA, seg_pos={0 0}, upload=523610]: Long tail cancellation due to optimal threshold Dec 30 12:49:42.416027156 (597h20m42.287110201s) [noise=true , address=45.91.116.4:4937 , piece_id=R2PNVVXRNFO6DVL64DCRHWJHV2CF7YFU744Q3QCQ3REVL7NFPV7Q, seg_pos={0 0}, upload=523610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 12:49:42.416034951 (597h20m42.287117986s) [noise=true , address=45.91.116.4:4937 , piece_id=R2PNVVXRNFO6DVL64DCRHWJHV2CF7YFU744Q3QCQ3REVL7NFPV7Q, seg_pos={0 0}, upload=523610]: Long tail cancellation due to optimal threshold Dec 30 12:49:42.416054017 (597h20m42.287137052s) [seg_pos={0 0}, upload=523610]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:49:42.416133308 (597h20m42.287216353s) [upload=523610]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:49:42.416138668 (597h20m42.287221703s) [upload=523610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:49:42.698342416 (597h20m42.569425451s) [upload=523610]: Done uploading segments... Dec 30 12:49:42.698361442 (597h20m42.569444477s) [upload=523610]: Upload finished. err: <nil> Duration: 6.189654366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X..................X.....X.X...X.........X....X........] duration: 43.021481089s trace: https://jaeger.dev.storj.io/trace/7c79692a0e48d03f FAIL (2025-12-30 12:51:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.334050703 (597h14m43.205133748s) [noise=true , address=145.79.227.30:4764 , piece_id=4LD37FRWYCHBWKBNI4UA5LOQZVZXXFSCRDF4RFNRTAZ77SJACSHQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.334530106 (597h14m43.205613172s) [noise=true , address=193.149.164.143:4696 , piece_id=NPJQKFXDIRS6JXAE6S2HXX7T7NJO6TA5Y42YHPWJDIYWM3L2NXVA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.335771409 (597h14m43.206854445s) [noise=true , address=77.90.31.13:4642 , piece_id=N4XIGOZ7D7YOKBH4POUBYTSUAHJDDUFI6P3ZMSLCMODH5K4EACKQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.336450683 (597h14m43.207533718s) [noise=true , address=145.79.227.43:4165 , piece_id=427IGEZITJ43QDDVMEKYO3SZNHHV32424DXZQLOIYY65G354TF7A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.337643555 (597h14m43.20872659s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.338109913 (597h14m43.209192958s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.338722419 (597h14m43.209805454s) [noise=true , address=77.90.31.12:4613 , piece_id=TL5RRD6XZBWKQGUNBKJPNNQWITLR272F6JGWTQ4CXB463QP2AJIA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.339679452 (597h14m43.210762497s) [noise=true , address=145.79.227.49:4349 , piece_id=4PA62EAXLAEXUXTJTPMT2V5AGU5EBERMO4QTJOUG6ENWJVE7XWCQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340043454 (597h14m43.21112649s) [noise=true , address=145.79.227.44:4200 , piece_id=URPS5E657KF54BEB4W75OY6BCNLRX6D624ZGX5IIUHXG7GRTS6EA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340456090 (597h14m43.211539125s) [noise=true , address=145.79.227.14:4868 , piece_id=Z3M7ZOHCDZYQJHFTD6CLGFRWDR2CSFPZ4RMUXRNBDAD3R4RAKG5A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340824472 (597h14m43.211907507s) [noise=true , address=145.79.227.34:4936 , piece_id=PIXFWRWIFRUGXOEF3VQP2LIM5Z7LBA2J4RT722RWEIRRLBZUCYIA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.341197592 (597h14m43.212280637s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.342542824 (597h14m43.213625869s) [noise=true , address=145.79.227.40:4071 , piece_id=S64U3WUWWFOGMMIFM7IOVB3QLNQVT3VNS63B4CMNNARFUBHK6YBQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.342936153 (597h14m43.214019188s) [noise=true , address=145.79.227.29:4963 , piece_id=N7NWRYQZD6XWPCGXD24LMTTATSFKTA4MZAIIRO6PY4GFGMIMK7TQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.343360019 (597h14m43.214443055s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:44.797759913 (597h14m44.668842948s) [upload=523526]: Next returned nil segment Dec 30 12:43:44.797916772 (597h14m44.668999837s) [upload=523526]: Waiting for error group managing segments... Dec 30 12:43:45.194916187 (597h14m45.065999222s) [noise=true , address=145.79.227.34:4936 , piece_id=PIXFWRWIFRUGXOEF3VQP2LIM5Z7LBA2J4RT722RWEIRRLBZUCYIA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.197273356 (597h14m45.068356391s) [noise=true , address=145.79.227.14:4868 , piece_id=Z3M7ZOHCDZYQJHFTD6CLGFRWDR2CSFPZ4RMUXRNBDAD3R4RAKG5A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.257722380 (597h14m45.128805425s) [noise=true , address=145.79.227.43:4165 , piece_id=427IGEZITJ43QDDVMEKYO3SZNHHV32424DXZQLOIYY65G354TF7A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.266885017 (597h14m45.137968052s) [noise=true , address=145.79.227.11:4739 , piece_id=K7CTKKOGLA3QOPHJULOKBOTZLBAC2IJRLQHYWCTRXWTEQSXHQXKA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.515366662 (597h14m45.386449698s) [noise=true , address=77.90.31.13:4642 , piece_id=N4XIGOZ7D7YOKBH4POUBYTSUAHJDDUFI6P3ZMSLCMODH5K4EACKQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.516015517 (597h14m45.387098552s) [noise=true , address=77.90.31.12:4613 , piece_id=TL5RRD6XZBWKQGUNBKJPNNQWITLR272F6JGWTQ4CXB463QP2AJIA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.675717965 (597h14m45.54680101s) [noise=true , address=45.91.116.9:4285 , piece_id=OLUYACDDTMFYAM3Y6YXDD7NVFRDF2HY7YNJVNXVEZGIZUOMMHMNA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.079843199 (597h14m45.950926244s) [noise=true , address=145.79.227.44:4200 , piece_id=URPS5E657KF54BEB4W75OY6BCNLRX6D624ZGX5IIUHXG7GRTS6EA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.107010909 (597h14m45.978093944s) [noise=true , address=145.79.227.49:4349 , piece_id=4PA62EAXLAEXUXTJTPMT2V5AGU5EBERMO4QTJOUG6ENWJVE7XWCQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.115841685 (597h14m45.98692472s) [noise=true , address=145.79.227.30:4764 , piece_id=4LD37FRWYCHBWKBNI4UA5LOQZVZXXFSCRDF4RFNRTAZ77SJACSHQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.146727476 (597h14m46.017810521s) [noise=true , address=145.79.227.29:4963 , piece_id=N7NWRYQZD6XWPCGXD24LMTTATSFKTA4MZAIIRO6PY4GFGMIMK7TQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.181192753 (597h14m46.052275788s) [noise=true , address=145.79.227.40:4071 , piece_id=S64U3WUWWFOGMMIFM7IOVB3QLNQVT3VNS63B4CMNNARFUBHK6YBQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.181465864 (597h14m46.052548899s) [noise=true , address=193.149.164.143:4696 , piece_id=NPJQKFXDIRS6JXAE6S2HXX7T7NJO6TA5Y42YHPWJDIYWM3L2NXVA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.299951046 (597h14m46.171034081s) [noise=true , address=64.49.13.26:4954 , piece_id=7TTXO2IQMBLL7CG2LQLN3PZELX7GFVUAQTPMMLPOK3DBHESQ6LZQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.299969181 (597h14m46.171052216s) [seg_pos={0 0}, upload=523526]: Segment reached optimal threshold of 14 pieces. Dec 30 12:43:46.300230178 (597h14m46.171313223s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection Dec 30 12:43:46.300237010 (597h14m46.171320045s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300243253 (597h14m46.171326288s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 30 12:43:46.300258882 (597h14m46.171341917s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300357881 (597h14m46.171440916s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Dec 30 12:43:46.300364003 (597h14m46.171447038s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300503578 (597h14m46.171586613s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH): protocol: read tcp 188.166.225.170:57434->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 12:43:46.300510040 (597h14m46.171593075s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300526192 (597h14m46.171609227s) [seg_pos={0 0}, upload=523526]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:43:46.300589201 (597h14m46.171672236s) [upload=523526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:43:46.300593630 (597h14m46.171676665s) [upload=523526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:43:46.605263857 (597h14m46.476346902s) [upload=523526]: Done uploading segments... Dec 30 12:43:46.605286911 (597h14m46.476369946s) [upload=523526]: Upload finished. err: <nil> Duration: 43.021499675s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X........X...] duration: 15.71637788s trace: https://jaeger.dev.storj.io/trace/5287a23df1c13acc FAIL (2025-12-30 12:51:32 UTC) orbiter warning: context deadline exceeded Duration: 15.716403538s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.......................X..] duration: 14.255775736s trace: https://jaeger.dev.storj.io/trace/2de4b260b3332c21 FAIL (2025-12-30 12:50:53 UTC) orbiter warning: context deadline exceeded Duration: 14.255795195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X........X..] duration: 12.95015872s trace: https://jaeger.dev.storj.io/trace/5287a23df1c13acc FAIL (2025-12-30 12:50:52 UTC) orbiter warning: context deadline exceeded Duration: 12.950184298s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [.X.....X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXX] duration: 5.001744203s trace: https://jaeger.dev.storj.io/trace/21f4d875299725da FAIL (2025-12-30 12:50:52 UTC) orbiter warning: context deadline exceeded Duration: 5.001762197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X..................X.....X.X...X.........X....X.......] duration: 39.45152533s trace: https://jaeger.dev.storj.io/trace/7c79692a0e48d03f FAIL (2025-12-30 12:50:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.334050703 (597h14m43.205133748s) [noise=true , address=145.79.227.30:4764 , piece_id=4LD37FRWYCHBWKBNI4UA5LOQZVZXXFSCRDF4RFNRTAZ77SJACSHQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.334530106 (597h14m43.205613172s) [noise=true , address=193.149.164.143:4696 , piece_id=NPJQKFXDIRS6JXAE6S2HXX7T7NJO6TA5Y42YHPWJDIYWM3L2NXVA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.335771409 (597h14m43.206854445s) [noise=true , address=77.90.31.13:4642 , piece_id=N4XIGOZ7D7YOKBH4POUBYTSUAHJDDUFI6P3ZMSLCMODH5K4EACKQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.336450683 (597h14m43.207533718s) [noise=true , address=145.79.227.43:4165 , piece_id=427IGEZITJ43QDDVMEKYO3SZNHHV32424DXZQLOIYY65G354TF7A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.337643555 (597h14m43.20872659s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.338109913 (597h14m43.209192958s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.338722419 (597h14m43.209805454s) [noise=true , address=77.90.31.12:4613 , piece_id=TL5RRD6XZBWKQGUNBKJPNNQWITLR272F6JGWTQ4CXB463QP2AJIA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.339679452 (597h14m43.210762497s) [noise=true , address=145.79.227.49:4349 , piece_id=4PA62EAXLAEXUXTJTPMT2V5AGU5EBERMO4QTJOUG6ENWJVE7XWCQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340043454 (597h14m43.21112649s) [noise=true , address=145.79.227.44:4200 , piece_id=URPS5E657KF54BEB4W75OY6BCNLRX6D624ZGX5IIUHXG7GRTS6EA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340456090 (597h14m43.211539125s) [noise=true , address=145.79.227.14:4868 , piece_id=Z3M7ZOHCDZYQJHFTD6CLGFRWDR2CSFPZ4RMUXRNBDAD3R4RAKG5A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340824472 (597h14m43.211907507s) [noise=true , address=145.79.227.34:4936 , piece_id=PIXFWRWIFRUGXOEF3VQP2LIM5Z7LBA2J4RT722RWEIRRLBZUCYIA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.341197592 (597h14m43.212280637s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.342542824 (597h14m43.213625869s) [noise=true , address=145.79.227.40:4071 , piece_id=S64U3WUWWFOGMMIFM7IOVB3QLNQVT3VNS63B4CMNNARFUBHK6YBQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.342936153 (597h14m43.214019188s) [noise=true , address=145.79.227.29:4963 , piece_id=N7NWRYQZD6XWPCGXD24LMTTATSFKTA4MZAIIRO6PY4GFGMIMK7TQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.343360019 (597h14m43.214443055s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:44.797759913 (597h14m44.668842948s) [upload=523526]: Next returned nil segment Dec 30 12:43:44.797916772 (597h14m44.668999837s) [upload=523526]: Waiting for error group managing segments... Dec 30 12:43:45.194916187 (597h14m45.065999222s) [noise=true , address=145.79.227.34:4936 , piece_id=PIXFWRWIFRUGXOEF3VQP2LIM5Z7LBA2J4RT722RWEIRRLBZUCYIA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.197273356 (597h14m45.068356391s) [noise=true , address=145.79.227.14:4868 , piece_id=Z3M7ZOHCDZYQJHFTD6CLGFRWDR2CSFPZ4RMUXRNBDAD3R4RAKG5A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.257722380 (597h14m45.128805425s) [noise=true , address=145.79.227.43:4165 , piece_id=427IGEZITJ43QDDVMEKYO3SZNHHV32424DXZQLOIYY65G354TF7A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.266885017 (597h14m45.137968052s) [noise=true , address=145.79.227.11:4739 , piece_id=K7CTKKOGLA3QOPHJULOKBOTZLBAC2IJRLQHYWCTRXWTEQSXHQXKA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.515366662 (597h14m45.386449698s) [noise=true , address=77.90.31.13:4642 , piece_id=N4XIGOZ7D7YOKBH4POUBYTSUAHJDDUFI6P3ZMSLCMODH5K4EACKQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.516015517 (597h14m45.387098552s) [noise=true , address=77.90.31.12:4613 , piece_id=TL5RRD6XZBWKQGUNBKJPNNQWITLR272F6JGWTQ4CXB463QP2AJIA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.675717965 (597h14m45.54680101s) [noise=true , address=45.91.116.9:4285 , piece_id=OLUYACDDTMFYAM3Y6YXDD7NVFRDF2HY7YNJVNXVEZGIZUOMMHMNA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.079843199 (597h14m45.950926244s) [noise=true , address=145.79.227.44:4200 , piece_id=URPS5E657KF54BEB4W75OY6BCNLRX6D624ZGX5IIUHXG7GRTS6EA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.107010909 (597h14m45.978093944s) [noise=true , address=145.79.227.49:4349 , piece_id=4PA62EAXLAEXUXTJTPMT2V5AGU5EBERMO4QTJOUG6ENWJVE7XWCQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.115841685 (597h14m45.98692472s) [noise=true , address=145.79.227.30:4764 , piece_id=4LD37FRWYCHBWKBNI4UA5LOQZVZXXFSCRDF4RFNRTAZ77SJACSHQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.146727476 (597h14m46.017810521s) [noise=true , address=145.79.227.29:4963 , piece_id=N7NWRYQZD6XWPCGXD24LMTTATSFKTA4MZAIIRO6PY4GFGMIMK7TQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.181192753 (597h14m46.052275788s) [noise=true , address=145.79.227.40:4071 , piece_id=S64U3WUWWFOGMMIFM7IOVB3QLNQVT3VNS63B4CMNNARFUBHK6YBQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.181465864 (597h14m46.052548899s) [noise=true , address=193.149.164.143:4696 , piece_id=NPJQKFXDIRS6JXAE6S2HXX7T7NJO6TA5Y42YHPWJDIYWM3L2NXVA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.299951046 (597h14m46.171034081s) [noise=true , address=64.49.13.26:4954 , piece_id=7TTXO2IQMBLL7CG2LQLN3PZELX7GFVUAQTPMMLPOK3DBHESQ6LZQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.299969181 (597h14m46.171052216s) [seg_pos={0 0}, upload=523526]: Segment reached optimal threshold of 14 pieces. Dec 30 12:43:46.300230178 (597h14m46.171313223s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection Dec 30 12:43:46.300237010 (597h14m46.171320045s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300243253 (597h14m46.171326288s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 30 12:43:46.300258882 (597h14m46.171341917s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300357881 (597h14m46.171440916s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Dec 30 12:43:46.300364003 (597h14m46.171447038s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300503578 (597h14m46.171586613s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH): protocol: read tcp 188.166.225.170:57434->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 12:43:46.300510040 (597h14m46.171593075s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300526192 (597h14m46.171609227s) [seg_pos={0 0}, upload=523526]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:43:46.300589201 (597h14m46.171672236s) [upload=523526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:43:46.300593630 (597h14m46.171676665s) [upload=523526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:43:46.605263857 (597h14m46.476346902s) [upload=523526]: Done uploading segments... Dec 30 12:43:46.605286911 (597h14m46.476369946s) [upload=523526]: Upload finished. err: <nil> Duration: 39.451552272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X........X.] duration: 17.519303124s trace: https://jaeger.dev.storj.io/trace/5287a23df1c13acc FAIL (2025-12-30 12:50:33 UTC) orbiter warning: context deadline exceeded Duration: 17.51935918s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [..X.....X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.XX] duration: 5.001406s trace: https://jaeger.dev.storj.io/trace/70b76015acc28f5e FAIL (2025-12-30 12:50:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001430206s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.......................X.] duration: 22.389616599s trace: https://jaeger.dev.storj.io/trace/2de4b260b3332c21 FAIL (2025-12-30 12:50:30 UTC) orbiter warning: context deadline exceeded Duration: 22.389636998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....XX.XX....XX..X.......X.XX..X..X.XX...............X..X.X.] duration: 5.577058482s trace: https://jaeger.dev.storj.io/trace/320d45110f7ef5b6 FAIL (2025-12-30 12:50:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 12:49:39.837093183 (597h20m39.708176228s) [upload=523610]: Starting upload Dec 30 12:49:39.837180680 (597h20m39.708263725s) [upload=523610]: Done starting upload Dec 30 12:49:39.837271182 (597h20m39.708354217s) [upload=523610]: Uploading segments... Dec 30 12:49:39.837525856 (597h20m39.708608891s) [upload=523610]: Got next segment. Inline: false Dec 30 12:49:39.837598575 (597h20m39.70868161s) [upload=523610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:49:39.837604036 (597h20m39.708687081s) [upload=523610]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:49:40.079647152 (597h20m39.950730197s) [seg_pos={0 0}, upload=523610]: Begin upload segment... Dec 30 12:49:40.079689804 (597h20m39.950772839s) [seg_pos={0 0}, upload=523610]: Done begin upload segment. Dec 30 12:49:40.079714861 (597h20m39.950797896s) [noise=true , address=145.79.227.26:4653 , piece_id=XHJSJEABGBMUCWGQBK34SHF2LQMN5SVTREFVJK5R4BHDWL4UZEUQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.079853104 (597h20m39.950936159s) [noise=true , address=145.79.227.10:4401 , piece_id=LUUDBIPEIPQZK43MXNKYHT2L5G2J7DS7IZ352IV4O2OQOJAMY2TA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.080372202 (597h20m39.951455237s) [noise=true , address=77.90.31.17:4773 , piece_id=37B2HCJC3LXKWYYYGXLGDX3WD2B3THUSW3I7IBF4RKBO6UBK7LHA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.080795949 (597h20m39.951878994s) [noise=true , address=145.79.227.36:4517 , piece_id=CLDJH2NRDZTLKLF4VB53OX4MTMBZ5HWCRJE2HVTYIKHV7XX5NJFA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.081173869 (597h20m39.952256904s) [noise=true , address=145.79.227.51:4410 , piece_id=C7M2U67WHWPJN7KPPZKHAY5O7SYP5A777WN7HVOQYGDIN2X6UQUQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.081568982 (597h20m39.952652027s) [noise=true , address=145.79.227.41:4085 , piece_id=XWVPP5HFUN6QEE4OOA5CN6L666DXEKOCNETLS3OEUHB5EGPI2DSQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.082005152 (597h20m39.953088187s) [noise=true , address=145.79.227.52:4447 , piece_id=RP6HLWFCJFTSAK2GTWNTWWFMBJJJ7INKTJDQOEISHTIGDCIXOMHQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.082407448 (597h20m39.953490483s) [noise=true , address=145.79.227.45:4231 , piece_id=565BVE3ALKOSWAOQCW52WUPWB5BSYVWMT5JZWTF4TBNTMCZTJ7UQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.083239231 (597h20m39.954322276s) [noise=true , address=45.91.116.14:4189 , piece_id=IXCXLOA3SN633DBOPI6NMX3YHZ4B5A5OZQJGVBX64TYAGAAV3ALQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.083670193 (597h20m39.954753228s) [noise=true , address=145.79.227.5:4221 , piece_id=RG6KPTD5J6SSLT6XKRLXK7NYPN3YHGGM4GW6BXTVQRPUWY2NARLA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.084370256 (597h20m39.955453291s) [noise=true , address=77.90.31.19:4841 , piece_id=WO27AQ7XCOG4KD22L2Y46C6HIANJ7IZHQ7X4JZWW45UCVHJA4NTQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.085458187 (597h20m39.956541222s) [noise=true , address=145.79.227.23:4547 , piece_id=OWINTYKOFJCUH5OL2PY3GO56WZS2IXFG6IH5WXXTPXYEPFQ23UXA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.086559975 (597h20m39.95764301s) [noise=true , address=45.91.116.4:4937 , piece_id=R2PNVVXRNFO6DVL64DCRHWJHV2CF7YFU744Q3QCQ3REVL7NFPV7Q, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.088229825 (597h20m39.95931286s) [noise=true , address=145.79.227.30:4761 , piece_id=RG6XRIBU2DD27WV7YGFKFYDGS33JJKFK6WFBTL6FLOFJMT3V5ELQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.089374163 (597h20m39.960457238s) [noise=true , address=145.79.227.3:4151 , piece_id=D3CMHDX3CJJVPQTD5I2LRX2TDAXE64AF5GCWIBPIH653HCBEHHAA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.090547417 (597h20m39.961630453s) [noise=true , address=45.91.116.5:4299 , piece_id=XDIIO3XXSWY5HZE72C5L4OZCK25LWMJCTCFGAUECYJOY5YXK23TQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:41.785690441 (597h20m41.656773486s) [upload=523610]: Next returned nil segment Dec 30 12:49:41.785704808 (597h20m41.656787844s) [upload=523610]: Waiting for error group managing segments... Dec 30 12:49:42.068560217 (597h20m41.939643262s) [noise=true , address=145.79.227.52:4447 , piece_id=RP6HLWFCJFTSAK2GTWNTWWFMBJJJ7INKTJDQOEISHTIGDCIXOMHQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.074125804 (597h20m41.94520884s) [noise=true , address=145.79.227.5:4221 , piece_id=RG6KPTD5J6SSLT6XKRLXK7NYPN3YHGGM4GW6BXTVQRPUWY2NARLA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.074506950 (597h20m41.945589995s) [noise=true , address=145.79.227.51:4410 , piece_id=C7M2U67WHWPJN7KPPZKHAY5O7SYP5A777WN7HVOQYGDIN2X6UQUQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.074770032 (597h20m41.945853067s) [noise=true , address=145.79.227.36:4517 , piece_id=CLDJH2NRDZTLKLF4VB53OX4MTMBZ5HWCRJE2HVTYIKHV7XX5NJFA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.076453878 (597h20m41.947536913s) [noise=true , address=145.79.227.23:4547 , piece_id=OWINTYKOFJCUH5OL2PY3GO56WZS2IXFG6IH5WXXTPXYEPFQ23UXA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.094770926 (597h20m41.965853961s) [noise=true , address=145.79.227.45:4231 , piece_id=565BVE3ALKOSWAOQCW52WUPWB5BSYVWMT5JZWTF4TBNTMCZTJ7UQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.095653527 (597h20m41.966736572s) [noise=true , address=145.79.227.41:4085 , piece_id=XWVPP5HFUN6QEE4OOA5CN6L666DXEKOCNETLS3OEUHB5EGPI2DSQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.098377954 (597h20m41.969460989s) [noise=true , address=145.79.227.3:4151 , piece_id=D3CMHDX3CJJVPQTD5I2LRX2TDAXE64AF5GCWIBPIH653HCBEHHAA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.150555228 (597h20m42.021638273s) [noise=true , address=145.79.227.10:4401 , piece_id=LUUDBIPEIPQZK43MXNKYHT2L5G2J7DS7IZ352IV4O2OQOJAMY2TA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.176734807 (597h20m42.047817842s) [noise=true , address=145.79.227.26:4653 , piece_id=XHJSJEABGBMUCWGQBK34SHF2LQMN5SVTREFVJK5R4BHDWL4UZEUQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.257699486 (597h20m42.128782541s) [noise=true , address=77.90.31.19:4841 , piece_id=WO27AQ7XCOG4KD22L2Y46C6HIANJ7IZHQ7X4JZWW45UCVHJA4NTQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.408523477 (597h20m42.279606512s) [noise=true , address=45.91.116.14:4189 , piece_id=IXCXLOA3SN633DBOPI6NMX3YHZ4B5A5OZQJGVBX64TYAGAAV3ALQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.415667241 (597h20m42.286750276s) [noise=true , address=45.91.116.5:4299 , piece_id=XDIIO3XXSWY5HZE72C5L4OZCK25LWMJCTCFGAUECYJOY5YXK23TQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.415691096 (597h20m42.286774131s) [seg_pos={0 0}, upload=523610]: Segment reached optimal threshold of 13 pieces. Dec 30 12:49:42.415868994 (597h20m42.286952039s) [noise=true , address=145.79.227.30:4761 , piece_id=RG6XRIBU2DD27WV7YGFKFYDGS33JJKFK6WFBTL6FLOFJMT3V5ELQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 30 12:49:42.415875267 (597h20m42.286958302s) [noise=true , address=145.79.227.30:4761 , piece_id=RG6XRIBU2DD27WV7YGFKFYDGS33JJKFK6WFBTL6FLOFJMT3V5ELQ, seg_pos={0 0}, upload=523610]: Long tail cancellation due to optimal threshold Dec 30 12:49:42.416000425 (597h20m42.28708346s) [noise=true , address=77.90.31.17:4773 , piece_id=37B2HCJC3LXKWYYYGXLGDX3WD2B3THUSW3I7IBF4RKBO6UBK7LHA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:43136->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:43136->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43136->77.90.31.17:4773: use of closed network connection Dec 30 12:49:42.416015775 (597h20m42.28709882s) [noise=true , address=77.90.31.17:4773 , piece_id=37B2HCJC3LXKWYYYGXLGDX3WD2B3THUSW3I7IBF4RKBO6UBK7LHA, seg_pos={0 0}, upload=523610]: Long tail cancellation due to optimal threshold Dec 30 12:49:42.416027156 (597h20m42.287110201s) [noise=true , address=45.91.116.4:4937 , piece_id=R2PNVVXRNFO6DVL64DCRHWJHV2CF7YFU744Q3QCQ3REVL7NFPV7Q, seg_pos={0 0}, upload=523610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 12:49:42.416034951 (597h20m42.287117986s) [noise=true , address=45.91.116.4:4937 , piece_id=R2PNVVXRNFO6DVL64DCRHWJHV2CF7YFU744Q3QCQ3REVL7NFPV7Q, seg_pos={0 0}, upload=523610]: Long tail cancellation due to optimal threshold Dec 30 12:49:42.416054017 (597h20m42.287137052s) [seg_pos={0 0}, upload=523610]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:49:42.416133308 (597h20m42.287216353s) [upload=523610]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:49:42.416138668 (597h20m42.287221703s) [upload=523610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:49:42.698342416 (597h20m42.569425451s) [upload=523610]: Done uploading segments... Dec 30 12:49:42.698361442 (597h20m42.569444477s) [upload=523610]: Upload finished. err: <nil> Duration: 5.577083219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....XX.XX....XX..X.......X.XX..X..X.XX...............X..X.X] duration: 30.004803547s trace: https://jaeger.dev.storj.io/trace/320d45110f7ef5b6 FAIL (2025-12-30 12:50:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 12:49:39.837093183 (597h20m39.708176228s) [upload=523610]: Starting upload Dec 30 12:49:39.837180680 (597h20m39.708263725s) [upload=523610]: Done starting upload Dec 30 12:49:39.837271182 (597h20m39.708354217s) [upload=523610]: Uploading segments... Dec 30 12:49:39.837525856 (597h20m39.708608891s) [upload=523610]: Got next segment. Inline: false Dec 30 12:49:39.837598575 (597h20m39.70868161s) [upload=523610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:49:39.837604036 (597h20m39.708687081s) [upload=523610]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:49:40.079647152 (597h20m39.950730197s) [seg_pos={0 0}, upload=523610]: Begin upload segment... Dec 30 12:49:40.079689804 (597h20m39.950772839s) [seg_pos={0 0}, upload=523610]: Done begin upload segment. Dec 30 12:49:40.079714861 (597h20m39.950797896s) [noise=true , address=145.79.227.26:4653 , piece_id=XHJSJEABGBMUCWGQBK34SHF2LQMN5SVTREFVJK5R4BHDWL4UZEUQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.079853104 (597h20m39.950936159s) [noise=true , address=145.79.227.10:4401 , piece_id=LUUDBIPEIPQZK43MXNKYHT2L5G2J7DS7IZ352IV4O2OQOJAMY2TA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.080372202 (597h20m39.951455237s) [noise=true , address=77.90.31.17:4773 , piece_id=37B2HCJC3LXKWYYYGXLGDX3WD2B3THUSW3I7IBF4RKBO6UBK7LHA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.080795949 (597h20m39.951878994s) [noise=true , address=145.79.227.36:4517 , piece_id=CLDJH2NRDZTLKLF4VB53OX4MTMBZ5HWCRJE2HVTYIKHV7XX5NJFA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.081173869 (597h20m39.952256904s) [noise=true , address=145.79.227.51:4410 , piece_id=C7M2U67WHWPJN7KPPZKHAY5O7SYP5A777WN7HVOQYGDIN2X6UQUQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.081568982 (597h20m39.952652027s) [noise=true , address=145.79.227.41:4085 , piece_id=XWVPP5HFUN6QEE4OOA5CN6L666DXEKOCNETLS3OEUHB5EGPI2DSQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.082005152 (597h20m39.953088187s) [noise=true , address=145.79.227.52:4447 , piece_id=RP6HLWFCJFTSAK2GTWNTWWFMBJJJ7INKTJDQOEISHTIGDCIXOMHQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.082407448 (597h20m39.953490483s) [noise=true , address=145.79.227.45:4231 , piece_id=565BVE3ALKOSWAOQCW52WUPWB5BSYVWMT5JZWTF4TBNTMCZTJ7UQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.083239231 (597h20m39.954322276s) [noise=true , address=45.91.116.14:4189 , piece_id=IXCXLOA3SN633DBOPI6NMX3YHZ4B5A5OZQJGVBX64TYAGAAV3ALQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.083670193 (597h20m39.954753228s) [noise=true , address=145.79.227.5:4221 , piece_id=RG6KPTD5J6SSLT6XKRLXK7NYPN3YHGGM4GW6BXTVQRPUWY2NARLA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.084370256 (597h20m39.955453291s) [noise=true , address=77.90.31.19:4841 , piece_id=WO27AQ7XCOG4KD22L2Y46C6HIANJ7IZHQ7X4JZWW45UCVHJA4NTQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.085458187 (597h20m39.956541222s) [noise=true , address=145.79.227.23:4547 , piece_id=OWINTYKOFJCUH5OL2PY3GO56WZS2IXFG6IH5WXXTPXYEPFQ23UXA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.086559975 (597h20m39.95764301s) [noise=true , address=45.91.116.4:4937 , piece_id=R2PNVVXRNFO6DVL64DCRHWJHV2CF7YFU744Q3QCQ3REVL7NFPV7Q, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.088229825 (597h20m39.95931286s) [noise=true , address=145.79.227.30:4761 , piece_id=RG6XRIBU2DD27WV7YGFKFYDGS33JJKFK6WFBTL6FLOFJMT3V5ELQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.089374163 (597h20m39.960457238s) [noise=true , address=145.79.227.3:4151 , piece_id=D3CMHDX3CJJVPQTD5I2LRX2TDAXE64AF5GCWIBPIH653HCBEHHAA, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:40.090547417 (597h20m39.961630453s) [noise=true , address=45.91.116.5:4299 , piece_id=XDIIO3XXSWY5HZE72C5L4OZCK25LWMJCTCFGAUECYJOY5YXK23TQ, seg_pos={0 0}, upload=523610]: Uploading piece... Dec 30 12:49:41.785690441 (597h20m41.656773486s) [upload=523610]: Next returned nil segment Dec 30 12:49:41.785704808 (597h20m41.656787844s) [upload=523610]: Waiting for error group managing segments... Dec 30 12:49:42.068560217 (597h20m41.939643262s) [noise=true , address=145.79.227.52:4447 , piece_id=RP6HLWFCJFTSAK2GTWNTWWFMBJJJ7INKTJDQOEISHTIGDCIXOMHQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.074125804 (597h20m41.94520884s) [noise=true , address=145.79.227.5:4221 , piece_id=RG6KPTD5J6SSLT6XKRLXK7NYPN3YHGGM4GW6BXTVQRPUWY2NARLA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.074506950 (597h20m41.945589995s) [noise=true , address=145.79.227.51:4410 , piece_id=C7M2U67WHWPJN7KPPZKHAY5O7SYP5A777WN7HVOQYGDIN2X6UQUQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.074770032 (597h20m41.945853067s) [noise=true , address=145.79.227.36:4517 , piece_id=CLDJH2NRDZTLKLF4VB53OX4MTMBZ5HWCRJE2HVTYIKHV7XX5NJFA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.076453878 (597h20m41.947536913s) [noise=true , address=145.79.227.23:4547 , piece_id=OWINTYKOFJCUH5OL2PY3GO56WZS2IXFG6IH5WXXTPXYEPFQ23UXA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.094770926 (597h20m41.965853961s) [noise=true , address=145.79.227.45:4231 , piece_id=565BVE3ALKOSWAOQCW52WUPWB5BSYVWMT5JZWTF4TBNTMCZTJ7UQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.095653527 (597h20m41.966736572s) [noise=true , address=145.79.227.41:4085 , piece_id=XWVPP5HFUN6QEE4OOA5CN6L666DXEKOCNETLS3OEUHB5EGPI2DSQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.098377954 (597h20m41.969460989s) [noise=true , address=145.79.227.3:4151 , piece_id=D3CMHDX3CJJVPQTD5I2LRX2TDAXE64AF5GCWIBPIH653HCBEHHAA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.150555228 (597h20m42.021638273s) [noise=true , address=145.79.227.10:4401 , piece_id=LUUDBIPEIPQZK43MXNKYHT2L5G2J7DS7IZ352IV4O2OQOJAMY2TA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.176734807 (597h20m42.047817842s) [noise=true , address=145.79.227.26:4653 , piece_id=XHJSJEABGBMUCWGQBK34SHF2LQMN5SVTREFVJK5R4BHDWL4UZEUQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.257699486 (597h20m42.128782541s) [noise=true , address=77.90.31.19:4841 , piece_id=WO27AQ7XCOG4KD22L2Y46C6HIANJ7IZHQ7X4JZWW45UCVHJA4NTQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.408523477 (597h20m42.279606512s) [noise=true , address=45.91.116.14:4189 , piece_id=IXCXLOA3SN633DBOPI6NMX3YHZ4B5A5OZQJGVBX64TYAGAAV3ALQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.415667241 (597h20m42.286750276s) [noise=true , address=45.91.116.5:4299 , piece_id=XDIIO3XXSWY5HZE72C5L4OZCK25LWMJCTCFGAUECYJOY5YXK23TQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: <nil> Dec 30 12:49:42.415691096 (597h20m42.286774131s) [seg_pos={0 0}, upload=523610]: Segment reached optimal threshold of 13 pieces. Dec 30 12:49:42.415868994 (597h20m42.286952039s) [noise=true , address=145.79.227.30:4761 , piece_id=RG6XRIBU2DD27WV7YGFKFYDGS33JJKFK6WFBTL6FLOFJMT3V5ELQ, seg_pos={0 0}, upload=523610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 30 12:49:42.415875267 (597h20m42.286958302s) [noise=true , address=145.79.227.30:4761 , piece_id=RG6XRIBU2DD27WV7YGFKFYDGS33JJKFK6WFBTL6FLOFJMT3V5ELQ, seg_pos={0 0}, upload=523610]: Long tail cancellation due to optimal threshold Dec 30 12:49:42.416000425 (597h20m42.28708346s) [noise=true , address=77.90.31.17:4773 , piece_id=37B2HCJC3LXKWYYYGXLGDX3WD2B3THUSW3I7IBF4RKBO6UBK7LHA, seg_pos={0 0}, upload=523610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:43136->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:43136->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43136->77.90.31.17:4773: use of closed network connection Dec 30 12:49:42.416015775 (597h20m42.28709882s) [noise=true , address=77.90.31.17:4773 , piece_id=37B2HCJC3LXKWYYYGXLGDX3WD2B3THUSW3I7IBF4RKBO6UBK7LHA, seg_pos={0 0}, upload=523610]: Long tail cancellation due to optimal threshold Dec 30 12:49:42.416027156 (597h20m42.287110201s) [noise=true , address=45.91.116.4:4937 , piece_id=R2PNVVXRNFO6DVL64DCRHWJHV2CF7YFU744Q3QCQ3REVL7NFPV7Q, seg_pos={0 0}, upload=523610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 12:49:42.416034951 (597h20m42.287117986s) [noise=true , address=45.91.116.4:4937 , piece_id=R2PNVVXRNFO6DVL64DCRHWJHV2CF7YFU744Q3QCQ3REVL7NFPV7Q, seg_pos={0 0}, upload=523610]: Long tail cancellation due to optimal threshold Dec 30 12:49:42.416054017 (597h20m42.287137052s) [seg_pos={0 0}, upload=523610]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:49:42.416133308 (597h20m42.287216353s) [upload=523610]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:49:42.416138668 (597h20m42.287221703s) [upload=523610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:49:42.698342416 (597h20m42.569425451s) [upload=523610]: Done uploading segments... Dec 30 12:49:42.698361442 (597h20m42.569444477s) [upload=523610]: Upload finished. err: <nil> Duration: 30.004872039s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.......................X] duration: 1m0.002358242s trace: https://jaeger.dev.storj.io/trace/2de4b260b3332c21 FAIL (2025-12-30 12:50:07 UTC) orbiter warning: context deadline exceeded Duration: 1m0.002389573s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [...X.....X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.X] duration: 5.002077005s trace: https://jaeger.dev.storj.io/trace/20193ae49f63320 FAIL (2025-12-30 12:50:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002098577s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X........X] duration: 30.001066381s trace: https://jaeger.dev.storj.io/trace/5287a23df1c13acc FAIL (2025-12-30 12:50:00 UTC) orbiter warning: context deadline exceeded Duration: 30.001085147s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [X...X.....X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X.] duration: 3.739482798s trace: https://jaeger.dev.storj.io/trace/5c514cd0ef71431f FAIL (2025-12-30 12:49:45 UTC) orbiter warning: context deadline exceeded Duration: 3.739506563s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X..................X.....X.X...X.........X....X......] duration: 38.027890661s trace: https://jaeger.dev.storj.io/trace/7c79692a0e48d03f FAIL (2025-12-30 12:49:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.334050703 (597h14m43.205133748s) [noise=true , address=145.79.227.30:4764 , piece_id=4LD37FRWYCHBWKBNI4UA5LOQZVZXXFSCRDF4RFNRTAZ77SJACSHQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.334530106 (597h14m43.205613172s) [noise=true , address=193.149.164.143:4696 , piece_id=NPJQKFXDIRS6JXAE6S2HXX7T7NJO6TA5Y42YHPWJDIYWM3L2NXVA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.335771409 (597h14m43.206854445s) [noise=true , address=77.90.31.13:4642 , piece_id=N4XIGOZ7D7YOKBH4POUBYTSUAHJDDUFI6P3ZMSLCMODH5K4EACKQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.336450683 (597h14m43.207533718s) [noise=true , address=145.79.227.43:4165 , piece_id=427IGEZITJ43QDDVMEKYO3SZNHHV32424DXZQLOIYY65G354TF7A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.337643555 (597h14m43.20872659s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.338109913 (597h14m43.209192958s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.338722419 (597h14m43.209805454s) [noise=true , address=77.90.31.12:4613 , piece_id=TL5RRD6XZBWKQGUNBKJPNNQWITLR272F6JGWTQ4CXB463QP2AJIA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.339679452 (597h14m43.210762497s) [noise=true , address=145.79.227.49:4349 , piece_id=4PA62EAXLAEXUXTJTPMT2V5AGU5EBERMO4QTJOUG6ENWJVE7XWCQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340043454 (597h14m43.21112649s) [noise=true , address=145.79.227.44:4200 , piece_id=URPS5E657KF54BEB4W75OY6BCNLRX6D624ZGX5IIUHXG7GRTS6EA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340456090 (597h14m43.211539125s) [noise=true , address=145.79.227.14:4868 , piece_id=Z3M7ZOHCDZYQJHFTD6CLGFRWDR2CSFPZ4RMUXRNBDAD3R4RAKG5A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340824472 (597h14m43.211907507s) [noise=true , address=145.79.227.34:4936 , piece_id=PIXFWRWIFRUGXOEF3VQP2LIM5Z7LBA2J4RT722RWEIRRLBZUCYIA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.341197592 (597h14m43.212280637s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.342542824 (597h14m43.213625869s) [noise=true , address=145.79.227.40:4071 , piece_id=S64U3WUWWFOGMMIFM7IOVB3QLNQVT3VNS63B4CMNNARFUBHK6YBQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.342936153 (597h14m43.214019188s) [noise=true , address=145.79.227.29:4963 , piece_id=N7NWRYQZD6XWPCGXD24LMTTATSFKTA4MZAIIRO6PY4GFGMIMK7TQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.343360019 (597h14m43.214443055s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:44.797759913 (597h14m44.668842948s) [upload=523526]: Next returned nil segment Dec 30 12:43:44.797916772 (597h14m44.668999837s) [upload=523526]: Waiting for error group managing segments... Dec 30 12:43:45.194916187 (597h14m45.065999222s) [noise=true , address=145.79.227.34:4936 , piece_id=PIXFWRWIFRUGXOEF3VQP2LIM5Z7LBA2J4RT722RWEIRRLBZUCYIA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.197273356 (597h14m45.068356391s) [noise=true , address=145.79.227.14:4868 , piece_id=Z3M7ZOHCDZYQJHFTD6CLGFRWDR2CSFPZ4RMUXRNBDAD3R4RAKG5A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.257722380 (597h14m45.128805425s) [noise=true , address=145.79.227.43:4165 , piece_id=427IGEZITJ43QDDVMEKYO3SZNHHV32424DXZQLOIYY65G354TF7A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.266885017 (597h14m45.137968052s) [noise=true , address=145.79.227.11:4739 , piece_id=K7CTKKOGLA3QOPHJULOKBOTZLBAC2IJRLQHYWCTRXWTEQSXHQXKA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.515366662 (597h14m45.386449698s) [noise=true , address=77.90.31.13:4642 , piece_id=N4XIGOZ7D7YOKBH4POUBYTSUAHJDDUFI6P3ZMSLCMODH5K4EACKQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.516015517 (597h14m45.387098552s) [noise=true , address=77.90.31.12:4613 , piece_id=TL5RRD6XZBWKQGUNBKJPNNQWITLR272F6JGWTQ4CXB463QP2AJIA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.675717965 (597h14m45.54680101s) [noise=true , address=45.91.116.9:4285 , piece_id=OLUYACDDTMFYAM3Y6YXDD7NVFRDF2HY7YNJVNXVEZGIZUOMMHMNA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.079843199 (597h14m45.950926244s) [noise=true , address=145.79.227.44:4200 , piece_id=URPS5E657KF54BEB4W75OY6BCNLRX6D624ZGX5IIUHXG7GRTS6EA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.107010909 (597h14m45.978093944s) [noise=true , address=145.79.227.49:4349 , piece_id=4PA62EAXLAEXUXTJTPMT2V5AGU5EBERMO4QTJOUG6ENWJVE7XWCQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.115841685 (597h14m45.98692472s) [noise=true , address=145.79.227.30:4764 , piece_id=4LD37FRWYCHBWKBNI4UA5LOQZVZXXFSCRDF4RFNRTAZ77SJACSHQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.146727476 (597h14m46.017810521s) [noise=true , address=145.79.227.29:4963 , piece_id=N7NWRYQZD6XWPCGXD24LMTTATSFKTA4MZAIIRO6PY4GFGMIMK7TQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.181192753 (597h14m46.052275788s) [noise=true , address=145.79.227.40:4071 , piece_id=S64U3WUWWFOGMMIFM7IOVB3QLNQVT3VNS63B4CMNNARFUBHK6YBQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.181465864 (597h14m46.052548899s) [noise=true , address=193.149.164.143:4696 , piece_id=NPJQKFXDIRS6JXAE6S2HXX7T7NJO6TA5Y42YHPWJDIYWM3L2NXVA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.299951046 (597h14m46.171034081s) [noise=true , address=64.49.13.26:4954 , piece_id=7TTXO2IQMBLL7CG2LQLN3PZELX7GFVUAQTPMMLPOK3DBHESQ6LZQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.299969181 (597h14m46.171052216s) [seg_pos={0 0}, upload=523526]: Segment reached optimal threshold of 14 pieces. Dec 30 12:43:46.300230178 (597h14m46.171313223s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection Dec 30 12:43:46.300237010 (597h14m46.171320045s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300243253 (597h14m46.171326288s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 30 12:43:46.300258882 (597h14m46.171341917s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300357881 (597h14m46.171440916s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Dec 30 12:43:46.300364003 (597h14m46.171447038s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300503578 (597h14m46.171586613s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH): protocol: read tcp 188.166.225.170:57434->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 12:43:46.300510040 (597h14m46.171593075s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300526192 (597h14m46.171609227s) [seg_pos={0 0}, upload=523526]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:43:46.300589201 (597h14m46.171672236s) [upload=523526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:43:46.300593630 (597h14m46.171676665s) [upload=523526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:43:46.605263857 (597h14m46.476346902s) [upload=523526]: Done uploading segments... Dec 30 12:43:46.605286911 (597h14m46.476369946s) [upload=523526]: Upload finished. err: <nil> Duration: 38.027904287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......XX.XX....XX..X.......X.XX..X..X.XX...............X..X.] duration: 5.840416368s trace: https://jaeger.dev.storj.io/trace/2d6a2c34d8feb6fc FAIL (2025-12-30 12:49:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 12:48:48.142882352 (597h19m48.013965387s) [upload=523594]: Starting upload Dec 30 12:48:48.142973396 (597h19m48.014056431s) [upload=523594]: Done starting upload Dec 30 12:48:48.143028791 (597h19m48.014111836s) [upload=523594]: Uploading segments... Dec 30 12:48:48.143264510 (597h19m48.014347545s) [upload=523594]: Got next segment. Inline: false Dec 30 12:48:48.143351707 (597h19m48.014434742s) [upload=523594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:48:48.143357337 (597h19m48.014440382s) [upload=523594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:48:48.387939678 (597h19m48.259022723s) [seg_pos={0 0}, upload=523594]: Begin upload segment... Dec 30 12:48:48.387999011 (597h19m48.259082046s) [seg_pos={0 0}, upload=523594]: Done begin upload segment. Dec 30 12:48:48.388030962 (597h19m48.259113997s) [noise=true , address=77.90.31.19:4842 , piece_id=44BP5ECUAP7DVZV6TBHZYNNBJUDKLVPZECLDDVC235VXLBHND6TA, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.388046201 (597h19m48.259129236s) [noise=true , address=145.79.227.3:4151 , piece_id=NIWJT45BSJCRJ5PB3G2Z6LQJCFZUN6GJ445AZNIBLG3PLGSFXORQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.388577643 (597h19m48.259660688s) [noise=true , address=77.90.31.18:4808 , piece_id=KASEZBYKSV2ZGIPZPBIOSNB6GQIXAIBMGM4RKTJQTC6UCYGLPPDA, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.388997462 (597h19m48.260080507s) [noise=true , address=145.79.227.11:4730 , piece_id=FHHQKKTYQLJGXYZJRWX7WLRXHL77CQVMUK6DH3B4TAF3ZVT7OA3A, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.389483137 (597h19m48.260566182s) [noise=true , address=45.91.116.9:4263 , piece_id=E5BE366GHJCXZ2IFFJQP6QAAHOO3CO6I5FEKBFCLAZDJZLJSM3MQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.389908977 (597h19m48.260992022s) [noise=true , address=45.91.116.12:4227 , piece_id=3HSL5UFPTSBOLT7ITAKGZ75AARZ2MBZN2DVPU5BR3B5EU52CKCCA, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.390343485 (597h19m48.26142652s) [noise=true , address=145.79.227.52:4445 , piece_id=2QMJ2Z3Y5NJNRUGEKFOSTGCTI5NN4JFUELJNJM6U4RYT5KU5OEAA, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.390797259 (597h19m48.261880294s) [noise=true , address=45.91.116.13:4154 , piece_id=HSACYSEOIYFWQCBTEK574QOSAB6YEGJO2DZTQGA5JQWCCWO2UXVQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.391273537 (597h19m48.262356582s) [noise=true , address=77.90.31.15:4697 , piece_id=QQZKL3BPB6ORGQPZ3KRXYIMWHWBYZ3RTEH4Y7OF7LIZ5FEP5FJ7Q, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.392488340 (597h19m48.263571375s) [noise=true , address=145.79.227.40:4049 , piece_id=VDY6FSKY63LYWCODVA7M3G7VEAIS4OCQKNEXD4S6ESXX4672QE4A, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.394631280 (597h19m48.265714315s) [noise=true , address=77.90.31.21:4878 , piece_id=CEX7YOHJUOXRBUZQBEQ6YSG2BIUH7TQUYMVVYKTL6SYYWH3NINZQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.395838118 (597h19m48.266921153s) [noise=true , address=77.90.31.10:4518 , piece_id=MWTKCA7NOBXJ2TW3MLBTHI4MN6VVAAKNGSFULZL56MWW2HD6Z2SA, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.397021732 (597h19m48.268104767s) [noise=true , address=145.79.227.2:4115 , piece_id=3GIZZKHY34GYI63SCYT5HKZUPWF6NCDMFMNF7K7OTJRSRRID5CSQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.398297691 (597h19m48.269380726s) [noise=true , address=145.79.227.23:4547 , piece_id=QNQZZMYUMOWKJKS4EUK2AWR7OZGVX6KKEJYJGEV67D7JE2MX6EQQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.399344976 (597h19m48.270428011s) [noise=true , address=145.79.227.37:4566 , piece_id=MZUGSP3VXVFCFVQ5WUQ2G3KCFSCIALDWJDNZJ5NGTI62IIVBJZ6A, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.400363255 (597h19m48.27144629s) [noise=true , address=145.79.227.9:4366 , piece_id=CSYLPY44OTHENIO5D5ML7FEDUCXKGP5MOJ4JAYKKSGSEOESAG22Q, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:50.102423610 (597h19m49.973506655s) [upload=523594]: Next returned nil segment Dec 30 12:48:50.102437797 (597h19m49.973520832s) [upload=523594]: Waiting for error group managing segments... Dec 30 12:48:50.376830181 (597h19m50.247913226s) [noise=true , address=145.79.227.11:4730 , piece_id=FHHQKKTYQLJGXYZJRWX7WLRXHL77CQVMUK6DH3B4TAF3ZVT7OA3A, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.389313470 (597h19m50.260396515s) [noise=true , address=145.79.227.3:4151 , piece_id=NIWJT45BSJCRJ5PB3G2Z6LQJCFZUN6GJ445AZNIBLG3PLGSFXORQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.396897071 (597h19m50.267980106s) [noise=true , address=145.79.227.37:4566 , piece_id=MZUGSP3VXVFCFVQ5WUQ2G3KCFSCIALDWJDNZJ5NGTI62IIVBJZ6A, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.399062404 (597h19m50.270145439s) [noise=true , address=145.79.227.52:4445 , piece_id=2QMJ2Z3Y5NJNRUGEKFOSTGCTI5NN4JFUELJNJM6U4RYT5KU5OEAA, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.404503314 (597h19m50.275586349s) [noise=true , address=145.79.227.2:4115 , piece_id=3GIZZKHY34GYI63SCYT5HKZUPWF6NCDMFMNF7K7OTJRSRRID5CSQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.409600971 (597h19m50.280684007s) [noise=true , address=145.79.227.23:4547 , piece_id=QNQZZMYUMOWKJKS4EUK2AWR7OZGVX6KKEJYJGEV67D7JE2MX6EQQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.414629887 (597h19m50.285712922s) [noise=true , address=145.79.227.9:4366 , piece_id=CSYLPY44OTHENIO5D5ML7FEDUCXKGP5MOJ4JAYKKSGSEOESAG22Q, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.686944474 (597h19m50.558027509s) [noise=true , address=77.90.31.10:4518 , piece_id=MWTKCA7NOBXJ2TW3MLBTHI4MN6VVAAKNGSFULZL56MWW2HD6Z2SA, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.688196068 (597h19m50.559279103s) [noise=true , address=77.90.31.15:4697 , piece_id=QQZKL3BPB6ORGQPZ3KRXYIMWHWBYZ3RTEH4Y7OF7LIZ5FEP5FJ7Q, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.706438764 (597h19m50.577521799s) [noise=true , address=45.91.116.12:4227 , piece_id=3HSL5UFPTSBOLT7ITAKGZ75AARZ2MBZN2DVPU5BR3B5EU52CKCCA, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.813656833 (597h19m50.684739868s) [noise=true , address=45.91.116.13:4154 , piece_id=HSACYSEOIYFWQCBTEK574QOSAB6YEGJO2DZTQGA5JQWCCWO2UXVQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.850723875 (597h19m50.72180691s) [noise=true , address=45.91.116.9:4263 , piece_id=E5BE366GHJCXZ2IFFJQP6QAAHOO3CO6I5FEKBFCLAZDJZLJSM3MQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:51.002956498 (597h19m50.874039543s) [noise=true , address=77.90.31.18:4808 , piece_id=KASEZBYKSV2ZGIPZPBIOSNB6GQIXAIBMGM4RKTJQTC6UCYGLPPDA, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:51.002981636 (597h19m50.874064681s) [seg_pos={0 0}, upload=523594]: Segment reached optimal threshold of 13 pieces. Dec 30 12:48:51.003259315 (597h19m50.87434237s) [noise=true , address=145.79.227.40:4049 , piece_id=VDY6FSKY63LYWCODVA7M3G7VEAIS4OCQKNEXD4S6ESXX4672QE4A, seg_pos={0 0}, upload=523594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:48:51.003269874 (597h19m50.874352909s) [noise=true , address=145.79.227.40:4049 , piece_id=VDY6FSKY63LYWCODVA7M3G7VEAIS4OCQKNEXD4S6ESXX4672QE4A, seg_pos={0 0}, upload=523594]: Long tail cancellation due to optimal threshold Dec 30 12:48:51.003429769 (597h19m50.874512824s) [noise=true , address=77.90.31.19:4842 , piece_id=44BP5ECUAP7DVZV6TBHZYNNBJUDKLVPZECLDDVC235VXLBHND6TA, seg_pos={0 0}, upload=523594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 30 12:48:51.003460037 (597h19m50.874543092s) [noise=true , address=77.90.31.19:4842 , piece_id=44BP5ECUAP7DVZV6TBHZYNNBJUDKLVPZECLDDVC235VXLBHND6TA, seg_pos={0 0}, upload=523594]: Long tail cancellation due to optimal threshold Dec 30 12:48:51.003620552 (597h19m50.874703597s) [noise=true , address=77.90.31.21:4878 , piece_id=CEX7YOHJUOXRBUZQBEQ6YSG2BIUH7TQUYMVVYKTL6SYYWH3NINZQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 30 12:48:51.003630943 (597h19m50.874713988s) [noise=true , address=77.90.31.21:4878 , piece_id=CEX7YOHJUOXRBUZQBEQ6YSG2BIUH7TQUYMVVYKTL6SYYWH3NINZQ, seg_pos={0 0}, upload=523594]: Long tail cancellation due to optimal threshold Dec 30 12:48:51.003650910 (597h19m50.874733955s) [seg_pos={0 0}, upload=523594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:48:51.003733006 (597h19m50.874816051s) [upload=523594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:48:51.003739790 (597h19m50.874822835s) [upload=523594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:48:51.299045064 (597h19m51.170128109s) [upload=523594]: Done uploading segments... Dec 30 12:48:51.299069691 (597h19m51.170152736s) [upload=523594]: Upload finished. err: <nil> Duration: 5.840431596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X......XX.XX....XX..X.......X.XX..X..X.XX...............X..X] duration: 30.002786802s trace: https://jaeger.dev.storj.io/trace/2d6a2c34d8feb6fc FAIL (2025-12-30 12:49:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 12:48:48.142882352 (597h19m48.013965387s) [upload=523594]: Starting upload Dec 30 12:48:48.142973396 (597h19m48.014056431s) [upload=523594]: Done starting upload Dec 30 12:48:48.143028791 (597h19m48.014111836s) [upload=523594]: Uploading segments... Dec 30 12:48:48.143264510 (597h19m48.014347545s) [upload=523594]: Got next segment. Inline: false Dec 30 12:48:48.143351707 (597h19m48.014434742s) [upload=523594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:48:48.143357337 (597h19m48.014440382s) [upload=523594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:48:48.387939678 (597h19m48.259022723s) [seg_pos={0 0}, upload=523594]: Begin upload segment... Dec 30 12:48:48.387999011 (597h19m48.259082046s) [seg_pos={0 0}, upload=523594]: Done begin upload segment. Dec 30 12:48:48.388030962 (597h19m48.259113997s) [noise=true , address=77.90.31.19:4842 , piece_id=44BP5ECUAP7DVZV6TBHZYNNBJUDKLVPZECLDDVC235VXLBHND6TA, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.388046201 (597h19m48.259129236s) [noise=true , address=145.79.227.3:4151 , piece_id=NIWJT45BSJCRJ5PB3G2Z6LQJCFZUN6GJ445AZNIBLG3PLGSFXORQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.388577643 (597h19m48.259660688s) [noise=true , address=77.90.31.18:4808 , piece_id=KASEZBYKSV2ZGIPZPBIOSNB6GQIXAIBMGM4RKTJQTC6UCYGLPPDA, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.388997462 (597h19m48.260080507s) [noise=true , address=145.79.227.11:4730 , piece_id=FHHQKKTYQLJGXYZJRWX7WLRXHL77CQVMUK6DH3B4TAF3ZVT7OA3A, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.389483137 (597h19m48.260566182s) [noise=true , address=45.91.116.9:4263 , piece_id=E5BE366GHJCXZ2IFFJQP6QAAHOO3CO6I5FEKBFCLAZDJZLJSM3MQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.389908977 (597h19m48.260992022s) [noise=true , address=45.91.116.12:4227 , piece_id=3HSL5UFPTSBOLT7ITAKGZ75AARZ2MBZN2DVPU5BR3B5EU52CKCCA, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.390343485 (597h19m48.26142652s) [noise=true , address=145.79.227.52:4445 , piece_id=2QMJ2Z3Y5NJNRUGEKFOSTGCTI5NN4JFUELJNJM6U4RYT5KU5OEAA, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.390797259 (597h19m48.261880294s) [noise=true , address=45.91.116.13:4154 , piece_id=HSACYSEOIYFWQCBTEK574QOSAB6YEGJO2DZTQGA5JQWCCWO2UXVQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.391273537 (597h19m48.262356582s) [noise=true , address=77.90.31.15:4697 , piece_id=QQZKL3BPB6ORGQPZ3KRXYIMWHWBYZ3RTEH4Y7OF7LIZ5FEP5FJ7Q, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.392488340 (597h19m48.263571375s) [noise=true , address=145.79.227.40:4049 , piece_id=VDY6FSKY63LYWCODVA7M3G7VEAIS4OCQKNEXD4S6ESXX4672QE4A, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.394631280 (597h19m48.265714315s) [noise=true , address=77.90.31.21:4878 , piece_id=CEX7YOHJUOXRBUZQBEQ6YSG2BIUH7TQUYMVVYKTL6SYYWH3NINZQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.395838118 (597h19m48.266921153s) [noise=true , address=77.90.31.10:4518 , piece_id=MWTKCA7NOBXJ2TW3MLBTHI4MN6VVAAKNGSFULZL56MWW2HD6Z2SA, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.397021732 (597h19m48.268104767s) [noise=true , address=145.79.227.2:4115 , piece_id=3GIZZKHY34GYI63SCYT5HKZUPWF6NCDMFMNF7K7OTJRSRRID5CSQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.398297691 (597h19m48.269380726s) [noise=true , address=145.79.227.23:4547 , piece_id=QNQZZMYUMOWKJKS4EUK2AWR7OZGVX6KKEJYJGEV67D7JE2MX6EQQ, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.399344976 (597h19m48.270428011s) [noise=true , address=145.79.227.37:4566 , piece_id=MZUGSP3VXVFCFVQ5WUQ2G3KCFSCIALDWJDNZJ5NGTI62IIVBJZ6A, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:48.400363255 (597h19m48.27144629s) [noise=true , address=145.79.227.9:4366 , piece_id=CSYLPY44OTHENIO5D5ML7FEDUCXKGP5MOJ4JAYKKSGSEOESAG22Q, seg_pos={0 0}, upload=523594]: Uploading piece... Dec 30 12:48:50.102423610 (597h19m49.973506655s) [upload=523594]: Next returned nil segment Dec 30 12:48:50.102437797 (597h19m49.973520832s) [upload=523594]: Waiting for error group managing segments... Dec 30 12:48:50.376830181 (597h19m50.247913226s) [noise=true , address=145.79.227.11:4730 , piece_id=FHHQKKTYQLJGXYZJRWX7WLRXHL77CQVMUK6DH3B4TAF3ZVT7OA3A, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.389313470 (597h19m50.260396515s) [noise=true , address=145.79.227.3:4151 , piece_id=NIWJT45BSJCRJ5PB3G2Z6LQJCFZUN6GJ445AZNIBLG3PLGSFXORQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.396897071 (597h19m50.267980106s) [noise=true , address=145.79.227.37:4566 , piece_id=MZUGSP3VXVFCFVQ5WUQ2G3KCFSCIALDWJDNZJ5NGTI62IIVBJZ6A, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.399062404 (597h19m50.270145439s) [noise=true , address=145.79.227.52:4445 , piece_id=2QMJ2Z3Y5NJNRUGEKFOSTGCTI5NN4JFUELJNJM6U4RYT5KU5OEAA, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.404503314 (597h19m50.275586349s) [noise=true , address=145.79.227.2:4115 , piece_id=3GIZZKHY34GYI63SCYT5HKZUPWF6NCDMFMNF7K7OTJRSRRID5CSQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.409600971 (597h19m50.280684007s) [noise=true , address=145.79.227.23:4547 , piece_id=QNQZZMYUMOWKJKS4EUK2AWR7OZGVX6KKEJYJGEV67D7JE2MX6EQQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.414629887 (597h19m50.285712922s) [noise=true , address=145.79.227.9:4366 , piece_id=CSYLPY44OTHENIO5D5ML7FEDUCXKGP5MOJ4JAYKKSGSEOESAG22Q, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.686944474 (597h19m50.558027509s) [noise=true , address=77.90.31.10:4518 , piece_id=MWTKCA7NOBXJ2TW3MLBTHI4MN6VVAAKNGSFULZL56MWW2HD6Z2SA, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.688196068 (597h19m50.559279103s) [noise=true , address=77.90.31.15:4697 , piece_id=QQZKL3BPB6ORGQPZ3KRXYIMWHWBYZ3RTEH4Y7OF7LIZ5FEP5FJ7Q, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.706438764 (597h19m50.577521799s) [noise=true , address=45.91.116.12:4227 , piece_id=3HSL5UFPTSBOLT7ITAKGZ75AARZ2MBZN2DVPU5BR3B5EU52CKCCA, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.813656833 (597h19m50.684739868s) [noise=true , address=45.91.116.13:4154 , piece_id=HSACYSEOIYFWQCBTEK574QOSAB6YEGJO2DZTQGA5JQWCCWO2UXVQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:50.850723875 (597h19m50.72180691s) [noise=true , address=45.91.116.9:4263 , piece_id=E5BE366GHJCXZ2IFFJQP6QAAHOO3CO6I5FEKBFCLAZDJZLJSM3MQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:51.002956498 (597h19m50.874039543s) [noise=true , address=77.90.31.18:4808 , piece_id=KASEZBYKSV2ZGIPZPBIOSNB6GQIXAIBMGM4RKTJQTC6UCYGLPPDA, seg_pos={0 0}, upload=523594]: Done uploading piece. err: <nil> Dec 30 12:48:51.002981636 (597h19m50.874064681s) [seg_pos={0 0}, upload=523594]: Segment reached optimal threshold of 13 pieces. Dec 30 12:48:51.003259315 (597h19m50.87434237s) [noise=true , address=145.79.227.40:4049 , piece_id=VDY6FSKY63LYWCODVA7M3G7VEAIS4OCQKNEXD4S6ESXX4672QE4A, seg_pos={0 0}, upload=523594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:48:51.003269874 (597h19m50.874352909s) [noise=true , address=145.79.227.40:4049 , piece_id=VDY6FSKY63LYWCODVA7M3G7VEAIS4OCQKNEXD4S6ESXX4672QE4A, seg_pos={0 0}, upload=523594]: Long tail cancellation due to optimal threshold Dec 30 12:48:51.003429769 (597h19m50.874512824s) [noise=true , address=77.90.31.19:4842 , piece_id=44BP5ECUAP7DVZV6TBHZYNNBJUDKLVPZECLDDVC235VXLBHND6TA, seg_pos={0 0}, upload=523594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 30 12:48:51.003460037 (597h19m50.874543092s) [noise=true , address=77.90.31.19:4842 , piece_id=44BP5ECUAP7DVZV6TBHZYNNBJUDKLVPZECLDDVC235VXLBHND6TA, seg_pos={0 0}, upload=523594]: Long tail cancellation due to optimal threshold Dec 30 12:48:51.003620552 (597h19m50.874703597s) [noise=true , address=77.90.31.21:4878 , piece_id=CEX7YOHJUOXRBUZQBEQ6YSG2BIUH7TQUYMVVYKTL6SYYWH3NINZQ, seg_pos={0 0}, upload=523594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 30 12:48:51.003630943 (597h19m50.874713988s) [noise=true , address=77.90.31.21:4878 , piece_id=CEX7YOHJUOXRBUZQBEQ6YSG2BIUH7TQUYMVVYKTL6SYYWH3NINZQ, seg_pos={0 0}, upload=523594]: Long tail cancellation due to optimal threshold Dec 30 12:48:51.003650910 (597h19m50.874733955s) [seg_pos={0 0}, upload=523594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:48:51.003733006 (597h19m50.874816051s) [upload=523594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:48:51.003739790 (597h19m50.874822835s) [upload=523594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:48:51.299045064 (597h19m51.170128109s) [upload=523594]: Done uploading segments... Dec 30 12:48:51.299069691 (597h19m51.170152736s) [upload=523594]: Upload finished. err: <nil> Duration: 30.002807932s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 8.809780879s trace: https://jaeger.dev.storj.io/trace/6591f30a4b25b85d FAIL (2025-12-30 12:49:17 UTC) orbiter warning: context deadline exceeded Duration: 8.809808741s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X..................X.....X.X...X.........X....X.....] duration: 36.225171493s trace: https://jaeger.dev.storj.io/trace/7c79692a0e48d03f FAIL (2025-12-30 12:48:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.334050703 (597h14m43.205133748s) [noise=true , address=145.79.227.30:4764 , piece_id=4LD37FRWYCHBWKBNI4UA5LOQZVZXXFSCRDF4RFNRTAZ77SJACSHQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.334530106 (597h14m43.205613172s) [noise=true , address=193.149.164.143:4696 , piece_id=NPJQKFXDIRS6JXAE6S2HXX7T7NJO6TA5Y42YHPWJDIYWM3L2NXVA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.335771409 (597h14m43.206854445s) [noise=true , address=77.90.31.13:4642 , piece_id=N4XIGOZ7D7YOKBH4POUBYTSUAHJDDUFI6P3ZMSLCMODH5K4EACKQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.336450683 (597h14m43.207533718s) [noise=true , address=145.79.227.43:4165 , piece_id=427IGEZITJ43QDDVMEKYO3SZNHHV32424DXZQLOIYY65G354TF7A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.337643555 (597h14m43.20872659s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.338109913 (597h14m43.209192958s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.338722419 (597h14m43.209805454s) [noise=true , address=77.90.31.12:4613 , piece_id=TL5RRD6XZBWKQGUNBKJPNNQWITLR272F6JGWTQ4CXB463QP2AJIA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.339679452 (597h14m43.210762497s) [noise=true , address=145.79.227.49:4349 , piece_id=4PA62EAXLAEXUXTJTPMT2V5AGU5EBERMO4QTJOUG6ENWJVE7XWCQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340043454 (597h14m43.21112649s) [noise=true , address=145.79.227.44:4200 , piece_id=URPS5E657KF54BEB4W75OY6BCNLRX6D624ZGX5IIUHXG7GRTS6EA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340456090 (597h14m43.211539125s) [noise=true , address=145.79.227.14:4868 , piece_id=Z3M7ZOHCDZYQJHFTD6CLGFRWDR2CSFPZ4RMUXRNBDAD3R4RAKG5A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340824472 (597h14m43.211907507s) [noise=true , address=145.79.227.34:4936 , piece_id=PIXFWRWIFRUGXOEF3VQP2LIM5Z7LBA2J4RT722RWEIRRLBZUCYIA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.341197592 (597h14m43.212280637s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.342542824 (597h14m43.213625869s) [noise=true , address=145.79.227.40:4071 , piece_id=S64U3WUWWFOGMMIFM7IOVB3QLNQVT3VNS63B4CMNNARFUBHK6YBQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.342936153 (597h14m43.214019188s) [noise=true , address=145.79.227.29:4963 , piece_id=N7NWRYQZD6XWPCGXD24LMTTATSFKTA4MZAIIRO6PY4GFGMIMK7TQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.343360019 (597h14m43.214443055s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:44.797759913 (597h14m44.668842948s) [upload=523526]: Next returned nil segment Dec 30 12:43:44.797916772 (597h14m44.668999837s) [upload=523526]: Waiting for error group managing segments... Dec 30 12:43:45.194916187 (597h14m45.065999222s) [noise=true , address=145.79.227.34:4936 , piece_id=PIXFWRWIFRUGXOEF3VQP2LIM5Z7LBA2J4RT722RWEIRRLBZUCYIA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.197273356 (597h14m45.068356391s) [noise=true , address=145.79.227.14:4868 , piece_id=Z3M7ZOHCDZYQJHFTD6CLGFRWDR2CSFPZ4RMUXRNBDAD3R4RAKG5A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.257722380 (597h14m45.128805425s) [noise=true , address=145.79.227.43:4165 , piece_id=427IGEZITJ43QDDVMEKYO3SZNHHV32424DXZQLOIYY65G354TF7A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.266885017 (597h14m45.137968052s) [noise=true , address=145.79.227.11:4739 , piece_id=K7CTKKOGLA3QOPHJULOKBOTZLBAC2IJRLQHYWCTRXWTEQSXHQXKA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.515366662 (597h14m45.386449698s) [noise=true , address=77.90.31.13:4642 , piece_id=N4XIGOZ7D7YOKBH4POUBYTSUAHJDDUFI6P3ZMSLCMODH5K4EACKQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.516015517 (597h14m45.387098552s) [noise=true , address=77.90.31.12:4613 , piece_id=TL5RRD6XZBWKQGUNBKJPNNQWITLR272F6JGWTQ4CXB463QP2AJIA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.675717965 (597h14m45.54680101s) [noise=true , address=45.91.116.9:4285 , piece_id=OLUYACDDTMFYAM3Y6YXDD7NVFRDF2HY7YNJVNXVEZGIZUOMMHMNA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.079843199 (597h14m45.950926244s) [noise=true , address=145.79.227.44:4200 , piece_id=URPS5E657KF54BEB4W75OY6BCNLRX6D624ZGX5IIUHXG7GRTS6EA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.107010909 (597h14m45.978093944s) [noise=true , address=145.79.227.49:4349 , piece_id=4PA62EAXLAEXUXTJTPMT2V5AGU5EBERMO4QTJOUG6ENWJVE7XWCQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.115841685 (597h14m45.98692472s) [noise=true , address=145.79.227.30:4764 , piece_id=4LD37FRWYCHBWKBNI4UA5LOQZVZXXFSCRDF4RFNRTAZ77SJACSHQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.146727476 (597h14m46.017810521s) [noise=true , address=145.79.227.29:4963 , piece_id=N7NWRYQZD6XWPCGXD24LMTTATSFKTA4MZAIIRO6PY4GFGMIMK7TQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.181192753 (597h14m46.052275788s) [noise=true , address=145.79.227.40:4071 , piece_id=S64U3WUWWFOGMMIFM7IOVB3QLNQVT3VNS63B4CMNNARFUBHK6YBQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.181465864 (597h14m46.052548899s) [noise=true , address=193.149.164.143:4696 , piece_id=NPJQKFXDIRS6JXAE6S2HXX7T7NJO6TA5Y42YHPWJDIYWM3L2NXVA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.299951046 (597h14m46.171034081s) [noise=true , address=64.49.13.26:4954 , piece_id=7TTXO2IQMBLL7CG2LQLN3PZELX7GFVUAQTPMMLPOK3DBHESQ6LZQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.299969181 (597h14m46.171052216s) [seg_pos={0 0}, upload=523526]: Segment reached optimal threshold of 14 pieces. Dec 30 12:43:46.300230178 (597h14m46.171313223s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection Dec 30 12:43:46.300237010 (597h14m46.171320045s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300243253 (597h14m46.171326288s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 30 12:43:46.300258882 (597h14m46.171341917s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300357881 (597h14m46.171440916s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Dec 30 12:43:46.300364003 (597h14m46.171447038s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300503578 (597h14m46.171586613s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH): protocol: read tcp 188.166.225.170:57434->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 12:43:46.300510040 (597h14m46.171593075s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300526192 (597h14m46.171609227s) [seg_pos={0 0}, upload=523526]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:43:46.300589201 (597h14m46.171672236s) [upload=523526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:43:46.300593630 (597h14m46.171676665s) [upload=523526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:43:46.605263857 (597h14m46.476346902s) [upload=523526]: Done uploading segments... Dec 30 12:43:46.605286911 (597h14m46.476369946s) [upload=523526]: Upload finished. err: <nil> Duration: 36.225187923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.X...X.....X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.X] duration: 5.001381861s trace: https://jaeger.dev.storj.io/trace/5c514cd0ef71431f FAIL (2025-12-30 12:48:59 UTC) orbiter warning: context deadline exceeded Duration: 5.00139725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 17.127553734s trace: https://jaeger.dev.storj.io/trace/6591f30a4b25b85d FAIL (2025-12-30 12:48:57 UTC) orbiter warning: context deadline exceeded Duration: 17.127592618s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 17.10874733s trace: https://jaeger.dev.storj.io/trace/3a5c5684b0772fe FAIL (2025-12-30 12:48:34 UTC) orbiter warning: context deadline exceeded Duration: 17.108776174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [..X...X.....X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX.] duration: 5.000367061s trace: https://jaeger.dev.storj.io/trace/1fb7a526140ed5ed FAIL (2025-12-30 12:48:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000390324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X..................X.....X.X...X.........X....X....] duration: 47.737106789s trace: https://jaeger.dev.storj.io/trace/7c79692a0e48d03f FAIL (2025-12-30 12:48:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.334050703 (597h14m43.205133748s) [noise=true , address=145.79.227.30:4764 , piece_id=4LD37FRWYCHBWKBNI4UA5LOQZVZXXFSCRDF4RFNRTAZ77SJACSHQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.334530106 (597h14m43.205613172s) [noise=true , address=193.149.164.143:4696 , piece_id=NPJQKFXDIRS6JXAE6S2HXX7T7NJO6TA5Y42YHPWJDIYWM3L2NXVA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.335771409 (597h14m43.206854445s) [noise=true , address=77.90.31.13:4642 , piece_id=N4XIGOZ7D7YOKBH4POUBYTSUAHJDDUFI6P3ZMSLCMODH5K4EACKQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.336450683 (597h14m43.207533718s) [noise=true , address=145.79.227.43:4165 , piece_id=427IGEZITJ43QDDVMEKYO3SZNHHV32424DXZQLOIYY65G354TF7A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.337643555 (597h14m43.20872659s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.338109913 (597h14m43.209192958s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.338722419 (597h14m43.209805454s) [noise=true , address=77.90.31.12:4613 , piece_id=TL5RRD6XZBWKQGUNBKJPNNQWITLR272F6JGWTQ4CXB463QP2AJIA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.339679452 (597h14m43.210762497s) [noise=true , address=145.79.227.49:4349 , piece_id=4PA62EAXLAEXUXTJTPMT2V5AGU5EBERMO4QTJOUG6ENWJVE7XWCQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340043454 (597h14m43.21112649s) [noise=true , address=145.79.227.44:4200 , piece_id=URPS5E657KF54BEB4W75OY6BCNLRX6D624ZGX5IIUHXG7GRTS6EA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340456090 (597h14m43.211539125s) [noise=true , address=145.79.227.14:4868 , piece_id=Z3M7ZOHCDZYQJHFTD6CLGFRWDR2CSFPZ4RMUXRNBDAD3R4RAKG5A, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.340824472 (597h14m43.211907507s) [noise=true , address=145.79.227.34:4936 , piece_id=PIXFWRWIFRUGXOEF3VQP2LIM5Z7LBA2J4RT722RWEIRRLBZUCYIA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.341197592 (597h14m43.212280637s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.342542824 (597h14m43.213625869s) [noise=true , address=145.79.227.40:4071 , piece_id=S64U3WUWWFOGMMIFM7IOVB3QLNQVT3VNS63B4CMNNARFUBHK6YBQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.342936153 (597h14m43.214019188s) [noise=true , address=145.79.227.29:4963 , piece_id=N7NWRYQZD6XWPCGXD24LMTTATSFKTA4MZAIIRO6PY4GFGMIMK7TQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:43.343360019 (597h14m43.214443055s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Uploading piece... Dec 30 12:43:44.797759913 (597h14m44.668842948s) [upload=523526]: Next returned nil segment Dec 30 12:43:44.797916772 (597h14m44.668999837s) [upload=523526]: Waiting for error group managing segments... Dec 30 12:43:45.194916187 (597h14m45.065999222s) [noise=true , address=145.79.227.34:4936 , piece_id=PIXFWRWIFRUGXOEF3VQP2LIM5Z7LBA2J4RT722RWEIRRLBZUCYIA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.197273356 (597h14m45.068356391s) [noise=true , address=145.79.227.14:4868 , piece_id=Z3M7ZOHCDZYQJHFTD6CLGFRWDR2CSFPZ4RMUXRNBDAD3R4RAKG5A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.257722380 (597h14m45.128805425s) [noise=true , address=145.79.227.43:4165 , piece_id=427IGEZITJ43QDDVMEKYO3SZNHHV32424DXZQLOIYY65G354TF7A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.266885017 (597h14m45.137968052s) [noise=true , address=145.79.227.11:4739 , piece_id=K7CTKKOGLA3QOPHJULOKBOTZLBAC2IJRLQHYWCTRXWTEQSXHQXKA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.515366662 (597h14m45.386449698s) [noise=true , address=77.90.31.13:4642 , piece_id=N4XIGOZ7D7YOKBH4POUBYTSUAHJDDUFI6P3ZMSLCMODH5K4EACKQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.516015517 (597h14m45.387098552s) [noise=true , address=77.90.31.12:4613 , piece_id=TL5RRD6XZBWKQGUNBKJPNNQWITLR272F6JGWTQ4CXB463QP2AJIA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:45.675717965 (597h14m45.54680101s) [noise=true , address=45.91.116.9:4285 , piece_id=OLUYACDDTMFYAM3Y6YXDD7NVFRDF2HY7YNJVNXVEZGIZUOMMHMNA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.079843199 (597h14m45.950926244s) [noise=true , address=145.79.227.44:4200 , piece_id=URPS5E657KF54BEB4W75OY6BCNLRX6D624ZGX5IIUHXG7GRTS6EA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.107010909 (597h14m45.978093944s) [noise=true , address=145.79.227.49:4349 , piece_id=4PA62EAXLAEXUXTJTPMT2V5AGU5EBERMO4QTJOUG6ENWJVE7XWCQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.115841685 (597h14m45.98692472s) [noise=true , address=145.79.227.30:4764 , piece_id=4LD37FRWYCHBWKBNI4UA5LOQZVZXXFSCRDF4RFNRTAZ77SJACSHQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.146727476 (597h14m46.017810521s) [noise=true , address=145.79.227.29:4963 , piece_id=N7NWRYQZD6XWPCGXD24LMTTATSFKTA4MZAIIRO6PY4GFGMIMK7TQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.181192753 (597h14m46.052275788s) [noise=true , address=145.79.227.40:4071 , piece_id=S64U3WUWWFOGMMIFM7IOVB3QLNQVT3VNS63B4CMNNARFUBHK6YBQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.181465864 (597h14m46.052548899s) [noise=true , address=193.149.164.143:4696 , piece_id=NPJQKFXDIRS6JXAE6S2HXX7T7NJO6TA5Y42YHPWJDIYWM3L2NXVA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.299951046 (597h14m46.171034081s) [noise=true , address=64.49.13.26:4954 , piece_id=7TTXO2IQMBLL7CG2LQLN3PZELX7GFVUAQTPMMLPOK3DBHESQ6LZQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: <nil> Dec 30 12:43:46.299969181 (597h14m46.171052216s) [seg_pos={0 0}, upload=523526]: Segment reached optimal threshold of 14 pieces. Dec 30 12:43:46.300230178 (597h14m46.171313223s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48006->64.49.13.169:4715: use of closed network connection Dec 30 12:43:46.300237010 (597h14m46.171320045s) [noise=true , address=64.49.13.169:4715 , piece_id=SREL5IQTJH3QU3I4GNV43NELGXSTBOEILDELDS34AOSHRRYNPFXQ, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300243253 (597h14m46.171326288s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 30 12:43:46.300258882 (597h14m46.171341917s) [noise=true , address=77.90.31.21:4909 , piece_id=K2OLX2YAULNGTKGQNQKJIDRACKPEYAFXMZIMM4IMRUCWYDIJU64Q, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300357881 (597h14m46.171440916s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Dec 30 12:43:46.300364003 (597h14m46.171447038s) [noise=true , address=145.79.227.35:4486 , piece_id=GHGKVXQBO3QFMMRADAX6Y5L7GLJPHIMC5DPA7HTZIPT7PPERKC3A, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300503578 (597h14m46.171586613s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH): protocol: read tcp 188.166.225.170:57434->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 12:43:46.300510040 (597h14m46.171593075s) [noise=true , address=45.91.116.12:4236 , piece_id=3VHHD24NH6WXJLHKJDFQ62F52Q4DDZ7VLIO7HIC3EGWT64QAGTOA, seg_pos={0 0}, upload=523526]: Long tail cancellation due to optimal threshold Dec 30 12:43:46.300526192 (597h14m46.171609227s) [seg_pos={0 0}, upload=523526]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:43:46.300589201 (597h14m46.171672236s) [upload=523526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:43:46.300593630 (597h14m46.171676665s) [upload=523526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:43:46.605263857 (597h14m46.476346902s) [upload=523526]: Done uploading segments... Dec 30 12:43:46.605286911 (597h14m46.476369946s) [upload=523526]: Upload finished. err: <nil> Duration: 47.73714392s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 8.43276104s trace: https://jaeger.dev.storj.io/trace/6591f30a4b25b85d FAIL (2025-12-30 12:47:59 UTC) orbiter warning: context deadline exceeded Duration: 8.432781259s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX......XX.XX....XX..X.......X.XX..X..X.XX...............X..] duration: 6.077468009s trace: https://jaeger.dev.storj.io/trace/3a527aed9dd2907c FAIL (2025-12-30 12:47:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: YLJJVOQ2V3XHYFMARYWLXSCXQRUUX7NXHPRA5KWH3I5LTJFR63OA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 4EWH6HZG5KL5MXI5M2JOU6KNKC723JKVYNIRHUWFMRHP5FHUDCQA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JWNXOOLNNUSANJVGCY4KWI456HEDBXJUOLG5GGN3XYR4W7C4XDKA) context deadline exceeded ======LOGS====== Dec 30 12:45:23.014577131 (597h16m22.885660166s) [upload=523546]: Starting upload Dec 30 12:45:23.014667252 (597h16m22.885750297s) [upload=523546]: Done starting upload Dec 30 12:45:23.014745763 (597h16m22.885828808s) [upload=523546]: Uploading segments... Dec 30 12:45:23.014918852 (597h16m22.886001887s) [upload=523546]: Got next segment. Inline: false Dec 30 12:45:23.014978796 (597h16m22.886061841s) [upload=523546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:45:23.014984047 (597h16m22.886067092s) [upload=523546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:45:23.258590830 (597h16m23.129673865s) [seg_pos={0 0}, upload=523546]: Begin upload segment... Dec 30 12:45:23.258636166 (597h16m23.129719201s) [seg_pos={0 0}, upload=523546]: Done begin upload segment. Dec 30 12:45:23.258662566 (597h16m23.129745601s) [noise=true , address=145.79.227.13:4804 , piece_id=AUNT4XCWEICACOYWA7UGOSMDFZ7I2QRFLBRDKLW7D543ZEKR2ARA, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.258785962 (597h16m23.129868997s) [noise=true , address=145.79.227.6:4257 , piece_id=IPRW5CRII4YKHFG4YIURNJBUZGQUNKJ3X5AZQSINLRFBR56V2UQQ, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.259360626 (597h16m23.130443661s) [noise=true , address=45.91.116.11:4120 , piece_id=L5SXCQTQTO7SFMPSJMCNPSLOK3T22BXNHDETALGQSOXNJIMD72SQ, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.259776908 (597h16m23.130859953s) [noise=true , address=145.79.227.23:4547 , piece_id=U52VNLD7H6ECIVDZJ4DFZXMS7UVEEEXJRDQTBROMBZ3L2QFHLZ4Q, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.260168804 (597h16m23.131251839s) [noise=true , address=77.90.31.19:4844 , piece_id=YLJJVOQ2V3XHYFMARYWLXSCXQRUUX7NXHPRA5KWH3I5LTJFR63OA, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.260572844 (597h16m23.131655889s) [noise=true , address=45.91.116.12:4226 , piece_id=4EWH6HZG5KL5MXI5M2JOU6KNKC723JKVYNIRHUWFMRHP5FHUDCQA, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.261005387 (597h16m23.132088422s) [noise=true , address=145.79.227.11:4729 , piece_id=AXMPBNU6YBCG52DYXJGLHSXKNSPSXZV4HUQUIC6MV3X4MNDQ6M5Q, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.262364285 (597h16m23.13344733s) [noise=true , address=145.79.227.32:4834 , piece_id=YW63FC3XOR3MULPSXSXBR5RAHV6SJGI66OZP34Q6LWDLDMOKJCDA, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.262511515 (597h16m23.13359456s) [noise=true , address=145.79.227.26:4654 , piece_id=PBRHOFD2WQS5VMU7CMVRCHZ4TQ7ICOZJFPNWY6SER3EUYTVTBICA, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.263525826 (597h16m23.134608861s) [noise=true , address=145.79.227.16:4910 , piece_id=VP3G2XZUT2QWTIP2IATRENMKPBPEWTYEDZGZWWW3O7T5Q3ICF2PQ, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.263990692 (597h16m23.135073727s) [noise=true , address=45.91.116.13:4154 , piece_id=JWNXOOLNNUSANJVGCY4KWI456HEDBXJUOLG5GGN3XYR4W7C4XDKA, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.264792930 (597h16m23.135875965s) [noise=true , address=45.91.116.9:4262 , piece_id=S76YFS56B6YNZCHNAUQAWZVQZLR5V5T6UEEDVK5HQ4VKHRWWOXWA, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.266118063 (597h16m23.137201098s) [noise=true , address=145.79.227.9:4366 , piece_id=62JBEQCCORAQC4GEWIXRWLY6FYK6YPEIKRD6LU2RFECNBIMKD7QA, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.267205393 (597h16m23.138288428s) [noise=true , address=77.90.31.11:4556 , piece_id=J7YNU775DYIOLODC53M6JG565YFMK6PHOVZJ2CDSERUUQL2XNY4Q, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.267775239 (597h16m23.138858274s) [noise=true , address=145.79.227.51:4411 , piece_id=ZXCUNBZSN7DDGIC43R6BGZ5CKHHEADGT2IRFCJ34BJMUKLJU4DGA, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:23.270861254 (597h16m23.141944299s) [noise=true , address=145.79.227.31:4799 , piece_id=JPY6JBUZ33WHFRFJZPG2DF3RBA2YEKSMVVPL7DGRGJVXFK73XKIA, seg_pos={0 0}, upload=523546]: Uploading piece... Dec 30 12:45:24.861105665 (597h16m24.7321887s) [upload=523546]: Next returned nil segment Dec 30 12:45:24.861120283 (597h16m24.732203318s) [upload=523546]: Waiting for error group managing segments... Dec 30 12:45:25.241332804 (597h16m25.112415849s) [noise=true , address=145.79.227.6:4257 , piece_id=IPRW5CRII4YKHFG4YIURNJBUZGQUNKJ3X5AZQSINLRFBR56V2UQQ, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.248243312 (597h16m25.119326348s) [noise=true , address=145.79.227.11:4729 , piece_id=AXMPBNU6YBCG52DYXJGLHSXKNSPSXZV4HUQUIC6MV3X4MNDQ6M5Q, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.250965426 (597h16m25.122048471s) [noise=true , address=145.79.227.23:4547 , piece_id=U52VNLD7H6ECIVDZJ4DFZXMS7UVEEEXJRDQTBROMBZ3L2QFHLZ4Q, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.258418247 (597h16m25.129501282s) [noise=true , address=145.79.227.16:4910 , piece_id=VP3G2XZUT2QWTIP2IATRENMKPBPEWTYEDZGZWWW3O7T5Q3ICF2PQ, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.262408435 (597h16m25.13349148s) [noise=true , address=145.79.227.51:4411 , piece_id=ZXCUNBZSN7DDGIC43R6BGZ5CKHHEADGT2IRFCJ34BJMUKLJU4DGA, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.289131699 (597h16m25.160214745s) [noise=true , address=145.79.227.26:4654 , piece_id=PBRHOFD2WQS5VMU7CMVRCHZ4TQ7ICOZJFPNWY6SER3EUYTVTBICA, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.300210947 (597h16m25.171293982s) [noise=true , address=145.79.227.9:4366 , piece_id=62JBEQCCORAQC4GEWIXRWLY6FYK6YPEIKRD6LU2RFECNBIMKD7QA, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.301612285 (597h16m25.17269532s) [noise=true , address=145.79.227.13:4804 , piece_id=AUNT4XCWEICACOYWA7UGOSMDFZ7I2QRFLBRDKLW7D543ZEKR2ARA, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.331989748 (597h16m25.203072793s) [noise=true , address=145.79.227.31:4799 , piece_id=JPY6JBUZ33WHFRFJZPG2DF3RBA2YEKSMVVPL7DGRGJVXFK73XKIA, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.349014004 (597h16m25.22009704s) [noise=true , address=145.79.227.32:4834 , piece_id=YW63FC3XOR3MULPSXSXBR5RAHV6SJGI66OZP34Q6LWDLDMOKJCDA, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.567795628 (597h16m25.438878663s) [noise=true , address=45.91.116.12:4226 , piece_id=4EWH6HZG5KL5MXI5M2JOU6KNKC723JKVYNIRHUWFMRHP5FHUDCQA, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:25.741226054 (597h16m25.612309099s) [noise=true , address=45.91.116.13:4154 , piece_id=JWNXOOLNNUSANJVGCY4KWI456HEDBXJUOLG5GGN3XYR4W7C4XDKA, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:26.210608158 (597h16m26.081691203s) [noise=true , address=77.90.31.19:4844 , piece_id=YLJJVOQ2V3XHYFMARYWLXSCXQRUUX7NXHPRA5KWH3I5LTJFR63OA, seg_pos={0 0}, upload=523546]: Done uploading piece. err: <nil> Dec 30 12:45:26.210627744 (597h16m26.08171078s) [seg_pos={0 0}, upload=523546]: Segment reached optimal threshold of 13 pieces. Dec 30 12:45:26.210868593 (597h16m26.081951638s) [noise=true , address=45.91.116.11:4120 , piece_id=L5SXCQTQTO7SFMPSJMCNPSLOK3T22BXNHDETALGQSOXNJIMD72SQ, seg_pos={0 0}, upload=523546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 12:45:26.210875276 (597h16m26.081958321s) [noise=true , address=45.91.116.11:4120 , piece_id=L5SXCQTQTO7SFMPSJMCNPSLOK3T22BXNHDETALGQSOXNJIMD72SQ, seg_pos={0 0}, upload=523546]: Long tail cancellation due to optimal threshold Dec 30 12:45:26.211064647 (597h16m26.082147682s) [noise=true , address=45.91.116.9:4262 , piece_id=S76YFS56B6YNZCHNAUQAWZVQZLR5V5T6UEEDVK5HQ4VKHRWWOXWA, seg_pos={0 0}, upload=523546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 12:45:26.211071900 (597h16m26.082154946s) [noise=true , address=45.91.116.9:4262 , piece_id=S76YFS56B6YNZCHNAUQAWZVQZLR5V5T6UEEDVK5HQ4VKHRWWOXWA, seg_pos={0 0}, upload=523546]: Long tail cancellation due to optimal threshold Dec 30 12:45:26.211323640 (597h16m26.082406685s) [noise=true , address=77.90.31.11:4556 , piece_id=J7YNU775DYIOLODC53M6JG565YFMK6PHOVZJ2CDSERUUQL2XNY4Q, seg_pos={0 0}, upload=523546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:43530->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:43530->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43530->77.90.31.11:4556: use of closed network connection Dec 30 12:45:26.211343748 (597h16m26.082426783s) [noise=true , address=77.90.31.11:4556 , piece_id=J7YNU775DYIOLODC53M6JG565YFMK6PHOVZJ2CDSERUUQL2XNY4Q, seg_pos={0 0}, upload=523546]: Long tail cancellation due to optimal threshold Dec 30 12:45:26.211417879 (597h16m26.082500914s) [seg_pos={0 0}, upload=523546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:45:26.211534180 (597h16m26.082617225s) [upload=523546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:45:26.211542857 (597h16m26.082625892s) [upload=523546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:45:26.497872411 (597h16m26.368955456s) [upload=523546]: Done uploading segments... Dec 30 12:45:26.497892831 (597h16m26.368975866s) [upload=523546]: Upload finished. err: <nil> Duration: 6.077491494s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [...X...X.....X..X..X..XXX.XX.XXX...X.XXXXXXXXXXX.X.XXXXXXXXX] duration: 5.000396557s trace: https://jaeger.dev.storj.io/trace/1fb7a526140ed5ed FAIL (2025-12-30 12:47:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000418758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.78951738s trace: https://jaeger.dev.storj.io/trace/3a5c5684b0772fe FAIL (2025-12-30 12:47:25 UTC) orbiter warning: context deadline exceeded Duration: 6.789540954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 7.325073538s trace: https://jaeger.dev.storj.io/trace/6591f30a4b25b85d FAIL (2025-12-30 12:47:02 UTC) orbiter warning: context deadline exceeded Duration: 7.325104817s edge gateway us-select-1

1
...
11245
11246
11247
11248
11249
...
18138