Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.379787933s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:25:07 UTC) orbiter warning: context deadline exceeded Duration: 3.380213391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.199949795s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:25:02 UTC) orbiter warning: context deadline exceeded Duration: 6.199980063s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.....XXX.X......................X..X.XX.X.X..X..X.XXX....] duration: 5.941072548s trace: https://jaeger.dev.storj.io/trace/327144d892a2e847 FAIL (2025-12-26 08:25:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:21:55.122904853 (496h52m54.993987898s) [upload=436109]: Starting upload Dec 26 08:21:55.122979596 (496h52m54.994062641s) [upload=436109]: Done starting upload Dec 26 08:21:55.123079816 (496h52m54.994162851s) [upload=436109]: Uploading segments... Dec 26 08:21:55.123225254 (496h52m54.994308289s) [upload=436109]: Got next segment. Inline: false Dec 26 08:21:55.123288744 (496h52m54.994371789s) [upload=436109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:21:55.123295397 (496h52m54.994378442s) [upload=436109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:21:55.366353472 (496h52m55.237436507s) [seg_pos={0 0}, upload=436109]: Begin upload segment... Dec 26 08:21:55.366413947 (496h52m55.237496982s) [seg_pos={0 0}, upload=436109]: Done begin upload segment. Dec 26 08:21:55.366479612 (496h52m55.237562657s) [noise=true , address=145.79.227.46:4602 , piece_id=AXXAEY2OS66YPVQGRZWVSDRPRW5YZS4FUX2P3HDLDOLPSZ4ZLDWQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.366490302 (496h52m55.237573337s) [noise=true , address=145.79.227.15:4876 , piece_id=SROQM3MC4KIJWOZFQ6QHBEUR7M7OTPK7LPIVL4X537LOPVDMYDRQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.366558953 (496h52m55.237641998s) [noise=true , address=145.79.227.7:4293 , piece_id=7HWLWI2VDKBNNU4PYZKKU2MI7HV7SEYI3DM3J3AM3KNFGDCTSSZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.367158616 (496h52m55.238241681s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.367780569 (496h52m55.238863614s) [noise=true , address=145.79.227.29:4942 , piece_id=P3NEFDCHGNYCHKNREO3W24UT3Y2JYDD7DRITY6VIHOCZHTQJ6UPQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.368647229 (496h52m55.239730275s) [noise=true , address=77.90.31.11:4554 , piece_id=NMGYD2NEH7JO5RYM2QK5J4SNMKVY4DULAWFI4JKRJACRH5A3N7CQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.368743052 (496h52m55.239826087s) [noise=true , address=145.79.227.42:4121 , piece_id=OZVKKT3LHSXVT3QYCPEOB4R55S3XKXGWOQCS5VLNFUKSHDDY62HQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.369129478 (496h52m55.240212523s) [noise=true , address=145.79.227.48:4301 , piece_id=EARSRIMER6XVHC7XHZ7IENWTZWGB4GKPVGRT7UIJEZV2DG3YG6ZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.370169819 (496h52m55.241252864s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.370629143 (496h52m55.241712179s) [noise=true , address=145.79.227.52:4445 , piece_id=2EGLPGSSAEW3JZEV4VRAXUEKC5KPWLMZQM54W3YR2XEJ5GCJM3WA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.371364123 (496h52m55.242447158s) [noise=true , address=145.79.227.33:4871 , piece_id=FK73ULAL6LXSYR3XKIENPGLDLQIRM62J7QEMY6ZOGNIRKP3UVSTQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.371776829 (496h52m55.242859874s) [noise=true , address=145.79.227.41:4087 , piece_id=N5LKUFBGUQZYQBPDLY5LVQ5BEMBNEJ2YY4VRJHUMUJ4CT67M5JLA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.372642968 (496h52m55.243726013s) [noise=true , address=45.91.116.11:4119 , piece_id=XH3DOGXGF4WUOQ5GZDTDSPV3ZLZ2A5GIGCZ4HM3LUVLYJP4KT57A, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.373850708 (496h52m55.244933753s) [noise=true , address=145.79.227.16:4909 , piece_id=3RGSW2XXGBZCFJM2AVCIIH7PVRTL4NQ4SGRL6Q2Z4WDK72RXFNOQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.374568565 (496h52m55.24565161s) [noise=true , address=145.79.227.6:4258 , piece_id=UEHDWUH3EA4AXU66HQ2MVYT6S7PX5OPJICLDGCOJHV7SQQ4HGOGA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.375882367 (496h52m55.246965402s) [noise=true , address=45.91.116.10:4082 , piece_id=JHC2BTBQMFIU6NKGDI2X5KB3EYSSIRDKORV46SD7U2O2YKHRWILA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:57.092354651 (496h52m56.963437686s) [upload=436109]: Next returned nil segment Dec 26 08:21:57.092370131 (496h52m56.963453167s) [upload=436109]: Waiting for error group managing segments... Dec 26 08:21:57.377216509 (496h52m57.248299555s) [noise=true , address=145.79.227.48:4301 , piece_id=EARSRIMER6XVHC7XHZ7IENWTZWGB4GKPVGRT7UIJEZV2DG3YG6ZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.381681541 (496h52m57.252764586s) [noise=true , address=145.79.227.52:4445 , piece_id=2EGLPGSSAEW3JZEV4VRAXUEKC5KPWLMZQM54W3YR2XEJ5GCJM3WA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.385419328 (496h52m57.256502373s) [noise=true , address=145.79.227.16:4909 , piece_id=3RGSW2XXGBZCFJM2AVCIIH7PVRTL4NQ4SGRL6Q2Z4WDK72RXFNOQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.385481707 (496h52m57.256564752s) [noise=true , address=145.79.227.29:4942 , piece_id=P3NEFDCHGNYCHKNREO3W24UT3Y2JYDD7DRITY6VIHOCZHTQJ6UPQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.392282417 (496h52m57.263365472s) [noise=true , address=145.79.227.42:4121 , piece_id=OZVKKT3LHSXVT3QYCPEOB4R55S3XKXGWOQCS5VLNFUKSHDDY62HQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.392560887 (496h52m57.263643922s) [noise=true , address=145.79.227.41:4087 , piece_id=N5LKUFBGUQZYQBPDLY5LVQ5BEMBNEJ2YY4VRJHUMUJ4CT67M5JLA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.403916531 (496h52m57.274999576s) [noise=true , address=145.79.227.7:4293 , piece_id=7HWLWI2VDKBNNU4PYZKKU2MI7HV7SEYI3DM3J3AM3KNFGDCTSSZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.405504594 (496h52m57.276587629s) [noise=true , address=145.79.227.46:4602 , piece_id=AXXAEY2OS66YPVQGRZWVSDRPRW5YZS4FUX2P3HDLDOLPSZ4ZLDWQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.406169921 (496h52m57.277252956s) [noise=true , address=145.79.227.15:4876 , piece_id=SROQM3MC4KIJWOZFQ6QHBEUR7M7OTPK7LPIVL4X537LOPVDMYDRQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.411074500 (496h52m57.282157545s) [noise=true , address=145.79.227.6:4258 , piece_id=UEHDWUH3EA4AXU66HQ2MVYT6S7PX5OPJICLDGCOJHV7SQQ4HGOGA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.500676197 (496h52m57.371759242s) [noise=true , address=145.79.227.33:4871 , piece_id=FK73ULAL6LXSYR3XKIENPGLDLQIRM62J7QEMY6ZOGNIRKP3UVSTQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.743736989 (496h52m57.614820024s) [noise=true , address=45.91.116.11:4119 , piece_id=XH3DOGXGF4WUOQ5GZDTDSPV3ZLZ2A5GIGCZ4HM3LUVLYJP4KT57A, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.921737773 (496h52m57.792820808s) [noise=true , address=77.90.31.11:4554 , piece_id=NMGYD2NEH7JO5RYM2QK5J4SNMKVY4DULAWFI4JKRJACRH5A3N7CQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 08:21:58.076627130 (496h52m57.947710165s) [noise=true , address=45.91.116.10:4082 , piece_id=JHC2BTBQMFIU6NKGDI2X5KB3EYSSIRDKORV46SD7U2O2YKHRWILA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:58.076647839 (496h52m57.947730884s) [seg_pos={0 0}, upload=436109]: Segment reached optimal threshold of 13 pieces. Dec 26 08:21:58.076946869 (496h52m57.948029904s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 08:21:58.076954663 (496h52m57.948037699s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Long tail cancellation due to optimal threshold Dec 26 08:21:58.077160215 (496h52m57.94824326s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:53900->45.91.116.13:4155: use of closed network connection; conn closed: op Dec 26 08:21:58.077170945 (496h52m57.94825398s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Long tail cancellation due to optimal threshold Dec 26 08:21:58.077194941 (496h52m57.948277986s) [seg_pos={0 0}, upload=436109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:21:58.077334998 (496h52m57.948418043s) [upload=436109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:21:58.077342832 (496h52m57.948425868s) [upload=436109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:21:58.363192762 (496h52m58.234275797s) [upload=436109]: Done uploading segments... Dec 26 08:21:58.363243778 (496h52m58.234326813s) [upload=436109]: Upload finished. err: <nil> Duration: 5.941104639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.....XXX.X......................X..X.XX.X.X..X..X.XXX...] duration: 5.744163559s trace: https://jaeger.dev.storj.io/trace/327144d892a2e847 FAIL (2025-12-26 08:24:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:21:55.122904853 (496h52m54.993987898s) [upload=436109]: Starting upload Dec 26 08:21:55.122979596 (496h52m54.994062641s) [upload=436109]: Done starting upload Dec 26 08:21:55.123079816 (496h52m54.994162851s) [upload=436109]: Uploading segments... Dec 26 08:21:55.123225254 (496h52m54.994308289s) [upload=436109]: Got next segment. Inline: false Dec 26 08:21:55.123288744 (496h52m54.994371789s) [upload=436109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:21:55.123295397 (496h52m54.994378442s) [upload=436109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:21:55.366353472 (496h52m55.237436507s) [seg_pos={0 0}, upload=436109]: Begin upload segment... Dec 26 08:21:55.366413947 (496h52m55.237496982s) [seg_pos={0 0}, upload=436109]: Done begin upload segment. Dec 26 08:21:55.366479612 (496h52m55.237562657s) [noise=true , address=145.79.227.46:4602 , piece_id=AXXAEY2OS66YPVQGRZWVSDRPRW5YZS4FUX2P3HDLDOLPSZ4ZLDWQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.366490302 (496h52m55.237573337s) [noise=true , address=145.79.227.15:4876 , piece_id=SROQM3MC4KIJWOZFQ6QHBEUR7M7OTPK7LPIVL4X537LOPVDMYDRQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.366558953 (496h52m55.237641998s) [noise=true , address=145.79.227.7:4293 , piece_id=7HWLWI2VDKBNNU4PYZKKU2MI7HV7SEYI3DM3J3AM3KNFGDCTSSZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.367158616 (496h52m55.238241681s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.367780569 (496h52m55.238863614s) [noise=true , address=145.79.227.29:4942 , piece_id=P3NEFDCHGNYCHKNREO3W24UT3Y2JYDD7DRITY6VIHOCZHTQJ6UPQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.368647229 (496h52m55.239730275s) [noise=true , address=77.90.31.11:4554 , piece_id=NMGYD2NEH7JO5RYM2QK5J4SNMKVY4DULAWFI4JKRJACRH5A3N7CQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.368743052 (496h52m55.239826087s) [noise=true , address=145.79.227.42:4121 , piece_id=OZVKKT3LHSXVT3QYCPEOB4R55S3XKXGWOQCS5VLNFUKSHDDY62HQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.369129478 (496h52m55.240212523s) [noise=true , address=145.79.227.48:4301 , piece_id=EARSRIMER6XVHC7XHZ7IENWTZWGB4GKPVGRT7UIJEZV2DG3YG6ZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.370169819 (496h52m55.241252864s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.370629143 (496h52m55.241712179s) [noise=true , address=145.79.227.52:4445 , piece_id=2EGLPGSSAEW3JZEV4VRAXUEKC5KPWLMZQM54W3YR2XEJ5GCJM3WA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.371364123 (496h52m55.242447158s) [noise=true , address=145.79.227.33:4871 , piece_id=FK73ULAL6LXSYR3XKIENPGLDLQIRM62J7QEMY6ZOGNIRKP3UVSTQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.371776829 (496h52m55.242859874s) [noise=true , address=145.79.227.41:4087 , piece_id=N5LKUFBGUQZYQBPDLY5LVQ5BEMBNEJ2YY4VRJHUMUJ4CT67M5JLA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.372642968 (496h52m55.243726013s) [noise=true , address=45.91.116.11:4119 , piece_id=XH3DOGXGF4WUOQ5GZDTDSPV3ZLZ2A5GIGCZ4HM3LUVLYJP4KT57A, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.373850708 (496h52m55.244933753s) [noise=true , address=145.79.227.16:4909 , piece_id=3RGSW2XXGBZCFJM2AVCIIH7PVRTL4NQ4SGRL6Q2Z4WDK72RXFNOQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.374568565 (496h52m55.24565161s) [noise=true , address=145.79.227.6:4258 , piece_id=UEHDWUH3EA4AXU66HQ2MVYT6S7PX5OPJICLDGCOJHV7SQQ4HGOGA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.375882367 (496h52m55.246965402s) [noise=true , address=45.91.116.10:4082 , piece_id=JHC2BTBQMFIU6NKGDI2X5KB3EYSSIRDKORV46SD7U2O2YKHRWILA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:57.092354651 (496h52m56.963437686s) [upload=436109]: Next returned nil segment Dec 26 08:21:57.092370131 (496h52m56.963453167s) [upload=436109]: Waiting for error group managing segments... Dec 26 08:21:57.377216509 (496h52m57.248299555s) [noise=true , address=145.79.227.48:4301 , piece_id=EARSRIMER6XVHC7XHZ7IENWTZWGB4GKPVGRT7UIJEZV2DG3YG6ZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.381681541 (496h52m57.252764586s) [noise=true , address=145.79.227.52:4445 , piece_id=2EGLPGSSAEW3JZEV4VRAXUEKC5KPWLMZQM54W3YR2XEJ5GCJM3WA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.385419328 (496h52m57.256502373s) [noise=true , address=145.79.227.16:4909 , piece_id=3RGSW2XXGBZCFJM2AVCIIH7PVRTL4NQ4SGRL6Q2Z4WDK72RXFNOQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.385481707 (496h52m57.256564752s) [noise=true , address=145.79.227.29:4942 , piece_id=P3NEFDCHGNYCHKNREO3W24UT3Y2JYDD7DRITY6VIHOCZHTQJ6UPQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.392282417 (496h52m57.263365472s) [noise=true , address=145.79.227.42:4121 , piece_id=OZVKKT3LHSXVT3QYCPEOB4R55S3XKXGWOQCS5VLNFUKSHDDY62HQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.392560887 (496h52m57.263643922s) [noise=true , address=145.79.227.41:4087 , piece_id=N5LKUFBGUQZYQBPDLY5LVQ5BEMBNEJ2YY4VRJHUMUJ4CT67M5JLA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.403916531 (496h52m57.274999576s) [noise=true , address=145.79.227.7:4293 , piece_id=7HWLWI2VDKBNNU4PYZKKU2MI7HV7SEYI3DM3J3AM3KNFGDCTSSZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.405504594 (496h52m57.276587629s) [noise=true , address=145.79.227.46:4602 , piece_id=AXXAEY2OS66YPVQGRZWVSDRPRW5YZS4FUX2P3HDLDOLPSZ4ZLDWQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.406169921 (496h52m57.277252956s) [noise=true , address=145.79.227.15:4876 , piece_id=SROQM3MC4KIJWOZFQ6QHBEUR7M7OTPK7LPIVL4X537LOPVDMYDRQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.411074500 (496h52m57.282157545s) [noise=true , address=145.79.227.6:4258 , piece_id=UEHDWUH3EA4AXU66HQ2MVYT6S7PX5OPJICLDGCOJHV7SQQ4HGOGA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.500676197 (496h52m57.371759242s) [noise=true , address=145.79.227.33:4871 , piece_id=FK73ULAL6LXSYR3XKIENPGLDLQIRM62J7QEMY6ZOGNIRKP3UVSTQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.743736989 (496h52m57.614820024s) [noise=true , address=45.91.116.11:4119 , piece_id=XH3DOGXGF4WUOQ5GZDTDSPV3ZLZ2A5GIGCZ4HM3LUVLYJP4KT57A, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.921737773 (496h52m57.792820808s) [noise=true , address=77.90.31.11:4554 , piece_id=NMGYD2NEH7JO5RYM2QK5J4SNMKVY4DULAWFI4JKRJACRH5A3N7CQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 08:21:58.076627130 (496h52m57.947710165s) [noise=true , address=45.91.116.10:4082 , piece_id=JHC2BTBQMFIU6NKGDI2X5KB3EYSSIRDKORV46SD7U2O2YKHRWILA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:58.076647839 (496h52m57.947730884s) [seg_pos={0 0}, upload=436109]: Segment reached optimal threshold of 13 pieces. Dec 26 08:21:58.076946869 (496h52m57.948029904s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 08:21:58.076954663 (496h52m57.948037699s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Long tail cancellation due to optimal threshold Dec 26 08:21:58.077160215 (496h52m57.94824326s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:53900->45.91.116.13:4155: use of closed network connection; conn closed: op Dec 26 08:21:58.077170945 (496h52m57.94825398s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Long tail cancellation due to optimal threshold Dec 26 08:21:58.077194941 (496h52m57.948277986s) [seg_pos={0 0}, upload=436109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:21:58.077334998 (496h52m57.948418043s) [upload=436109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:21:58.077342832 (496h52m57.948425868s) [upload=436109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:21:58.363192762 (496h52m58.234275797s) [upload=436109]: Done uploading segments... Dec 26 08:21:58.363243778 (496h52m58.234326813s) [upload=436109]: Upload finished. err: <nil> Duration: 5.744186452s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 9.68210373s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:24:09 UTC) orbiter warning: context deadline exceeded Duration: 9.68213024s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.427854127s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:24:08 UTC) orbiter warning: context deadline exceeded Duration: 3.427874076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.491834467s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:23:45 UTC) orbiter warning: context deadline exceeded Duration: 6.491864022s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.....XXX.X......................X..X.XX.X.X..X..X.XXX..] duration: 5.755723742s trace: https://jaeger.dev.storj.io/trace/327144d892a2e847 FAIL (2025-12-26 08:23:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:21:55.122904853 (496h52m54.993987898s) [upload=436109]: Starting upload Dec 26 08:21:55.122979596 (496h52m54.994062641s) [upload=436109]: Done starting upload Dec 26 08:21:55.123079816 (496h52m54.994162851s) [upload=436109]: Uploading segments... Dec 26 08:21:55.123225254 (496h52m54.994308289s) [upload=436109]: Got next segment. Inline: false Dec 26 08:21:55.123288744 (496h52m54.994371789s) [upload=436109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:21:55.123295397 (496h52m54.994378442s) [upload=436109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:21:55.366353472 (496h52m55.237436507s) [seg_pos={0 0}, upload=436109]: Begin upload segment... Dec 26 08:21:55.366413947 (496h52m55.237496982s) [seg_pos={0 0}, upload=436109]: Done begin upload segment. Dec 26 08:21:55.366479612 (496h52m55.237562657s) [noise=true , address=145.79.227.46:4602 , piece_id=AXXAEY2OS66YPVQGRZWVSDRPRW5YZS4FUX2P3HDLDOLPSZ4ZLDWQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.366490302 (496h52m55.237573337s) [noise=true , address=145.79.227.15:4876 , piece_id=SROQM3MC4KIJWOZFQ6QHBEUR7M7OTPK7LPIVL4X537LOPVDMYDRQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.366558953 (496h52m55.237641998s) [noise=true , address=145.79.227.7:4293 , piece_id=7HWLWI2VDKBNNU4PYZKKU2MI7HV7SEYI3DM3J3AM3KNFGDCTSSZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.367158616 (496h52m55.238241681s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.367780569 (496h52m55.238863614s) [noise=true , address=145.79.227.29:4942 , piece_id=P3NEFDCHGNYCHKNREO3W24UT3Y2JYDD7DRITY6VIHOCZHTQJ6UPQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.368647229 (496h52m55.239730275s) [noise=true , address=77.90.31.11:4554 , piece_id=NMGYD2NEH7JO5RYM2QK5J4SNMKVY4DULAWFI4JKRJACRH5A3N7CQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.368743052 (496h52m55.239826087s) [noise=true , address=145.79.227.42:4121 , piece_id=OZVKKT3LHSXVT3QYCPEOB4R55S3XKXGWOQCS5VLNFUKSHDDY62HQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.369129478 (496h52m55.240212523s) [noise=true , address=145.79.227.48:4301 , piece_id=EARSRIMER6XVHC7XHZ7IENWTZWGB4GKPVGRT7UIJEZV2DG3YG6ZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.370169819 (496h52m55.241252864s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.370629143 (496h52m55.241712179s) [noise=true , address=145.79.227.52:4445 , piece_id=2EGLPGSSAEW3JZEV4VRAXUEKC5KPWLMZQM54W3YR2XEJ5GCJM3WA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.371364123 (496h52m55.242447158s) [noise=true , address=145.79.227.33:4871 , piece_id=FK73ULAL6LXSYR3XKIENPGLDLQIRM62J7QEMY6ZOGNIRKP3UVSTQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.371776829 (496h52m55.242859874s) [noise=true , address=145.79.227.41:4087 , piece_id=N5LKUFBGUQZYQBPDLY5LVQ5BEMBNEJ2YY4VRJHUMUJ4CT67M5JLA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.372642968 (496h52m55.243726013s) [noise=true , address=45.91.116.11:4119 , piece_id=XH3DOGXGF4WUOQ5GZDTDSPV3ZLZ2A5GIGCZ4HM3LUVLYJP4KT57A, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.373850708 (496h52m55.244933753s) [noise=true , address=145.79.227.16:4909 , piece_id=3RGSW2XXGBZCFJM2AVCIIH7PVRTL4NQ4SGRL6Q2Z4WDK72RXFNOQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.374568565 (496h52m55.24565161s) [noise=true , address=145.79.227.6:4258 , piece_id=UEHDWUH3EA4AXU66HQ2MVYT6S7PX5OPJICLDGCOJHV7SQQ4HGOGA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.375882367 (496h52m55.246965402s) [noise=true , address=45.91.116.10:4082 , piece_id=JHC2BTBQMFIU6NKGDI2X5KB3EYSSIRDKORV46SD7U2O2YKHRWILA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:57.092354651 (496h52m56.963437686s) [upload=436109]: Next returned nil segment Dec 26 08:21:57.092370131 (496h52m56.963453167s) [upload=436109]: Waiting for error group managing segments... Dec 26 08:21:57.377216509 (496h52m57.248299555s) [noise=true , address=145.79.227.48:4301 , piece_id=EARSRIMER6XVHC7XHZ7IENWTZWGB4GKPVGRT7UIJEZV2DG3YG6ZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.381681541 (496h52m57.252764586s) [noise=true , address=145.79.227.52:4445 , piece_id=2EGLPGSSAEW3JZEV4VRAXUEKC5KPWLMZQM54W3YR2XEJ5GCJM3WA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.385419328 (496h52m57.256502373s) [noise=true , address=145.79.227.16:4909 , piece_id=3RGSW2XXGBZCFJM2AVCIIH7PVRTL4NQ4SGRL6Q2Z4WDK72RXFNOQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.385481707 (496h52m57.256564752s) [noise=true , address=145.79.227.29:4942 , piece_id=P3NEFDCHGNYCHKNREO3W24UT3Y2JYDD7DRITY6VIHOCZHTQJ6UPQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.392282417 (496h52m57.263365472s) [noise=true , address=145.79.227.42:4121 , piece_id=OZVKKT3LHSXVT3QYCPEOB4R55S3XKXGWOQCS5VLNFUKSHDDY62HQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.392560887 (496h52m57.263643922s) [noise=true , address=145.79.227.41:4087 , piece_id=N5LKUFBGUQZYQBPDLY5LVQ5BEMBNEJ2YY4VRJHUMUJ4CT67M5JLA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.403916531 (496h52m57.274999576s) [noise=true , address=145.79.227.7:4293 , piece_id=7HWLWI2VDKBNNU4PYZKKU2MI7HV7SEYI3DM3J3AM3KNFGDCTSSZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.405504594 (496h52m57.276587629s) [noise=true , address=145.79.227.46:4602 , piece_id=AXXAEY2OS66YPVQGRZWVSDRPRW5YZS4FUX2P3HDLDOLPSZ4ZLDWQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.406169921 (496h52m57.277252956s) [noise=true , address=145.79.227.15:4876 , piece_id=SROQM3MC4KIJWOZFQ6QHBEUR7M7OTPK7LPIVL4X537LOPVDMYDRQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.411074500 (496h52m57.282157545s) [noise=true , address=145.79.227.6:4258 , piece_id=UEHDWUH3EA4AXU66HQ2MVYT6S7PX5OPJICLDGCOJHV7SQQ4HGOGA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.500676197 (496h52m57.371759242s) [noise=true , address=145.79.227.33:4871 , piece_id=FK73ULAL6LXSYR3XKIENPGLDLQIRM62J7QEMY6ZOGNIRKP3UVSTQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.743736989 (496h52m57.614820024s) [noise=true , address=45.91.116.11:4119 , piece_id=XH3DOGXGF4WUOQ5GZDTDSPV3ZLZ2A5GIGCZ4HM3LUVLYJP4KT57A, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.921737773 (496h52m57.792820808s) [noise=true , address=77.90.31.11:4554 , piece_id=NMGYD2NEH7JO5RYM2QK5J4SNMKVY4DULAWFI4JKRJACRH5A3N7CQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 08:21:58.076627130 (496h52m57.947710165s) [noise=true , address=45.91.116.10:4082 , piece_id=JHC2BTBQMFIU6NKGDI2X5KB3EYSSIRDKORV46SD7U2O2YKHRWILA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:58.076647839 (496h52m57.947730884s) [seg_pos={0 0}, upload=436109]: Segment reached optimal threshold of 13 pieces. Dec 26 08:21:58.076946869 (496h52m57.948029904s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 08:21:58.076954663 (496h52m57.948037699s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Long tail cancellation due to optimal threshold Dec 26 08:21:58.077160215 (496h52m57.94824326s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:53900->45.91.116.13:4155: use of closed network connection; conn closed: op Dec 26 08:21:58.077170945 (496h52m57.94825398s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Long tail cancellation due to optimal threshold Dec 26 08:21:58.077194941 (496h52m57.948277986s) [seg_pos={0 0}, upload=436109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:21:58.077334998 (496h52m57.948418043s) [upload=436109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:21:58.077342832 (496h52m57.948425868s) [upload=436109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:21:58.363192762 (496h52m58.234275797s) [upload=436109]: Done uploading segments... Dec 26 08:21:58.363243778 (496h52m58.234326813s) [upload=436109]: Upload finished. err: <nil> Duration: 5.755758319s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.434169453s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:23:10 UTC) orbiter warning: context deadline exceeded Duration: 3.434199861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X.....XXX.X......................X..X.XX.X.X..X..X.XXX.] duration: 6.017166475s trace: https://jaeger.dev.storj.io/trace/327144d892a2e847 FAIL (2025-12-26 08:23:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:21:55.122904853 (496h52m54.993987898s) [upload=436109]: Starting upload Dec 26 08:21:55.122979596 (496h52m54.994062641s) [upload=436109]: Done starting upload Dec 26 08:21:55.123079816 (496h52m54.994162851s) [upload=436109]: Uploading segments... Dec 26 08:21:55.123225254 (496h52m54.994308289s) [upload=436109]: Got next segment. Inline: false Dec 26 08:21:55.123288744 (496h52m54.994371789s) [upload=436109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:21:55.123295397 (496h52m54.994378442s) [upload=436109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:21:55.366353472 (496h52m55.237436507s) [seg_pos={0 0}, upload=436109]: Begin upload segment... Dec 26 08:21:55.366413947 (496h52m55.237496982s) [seg_pos={0 0}, upload=436109]: Done begin upload segment. Dec 26 08:21:55.366479612 (496h52m55.237562657s) [noise=true , address=145.79.227.46:4602 , piece_id=AXXAEY2OS66YPVQGRZWVSDRPRW5YZS4FUX2P3HDLDOLPSZ4ZLDWQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.366490302 (496h52m55.237573337s) [noise=true , address=145.79.227.15:4876 , piece_id=SROQM3MC4KIJWOZFQ6QHBEUR7M7OTPK7LPIVL4X537LOPVDMYDRQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.366558953 (496h52m55.237641998s) [noise=true , address=145.79.227.7:4293 , piece_id=7HWLWI2VDKBNNU4PYZKKU2MI7HV7SEYI3DM3J3AM3KNFGDCTSSZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.367158616 (496h52m55.238241681s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.367780569 (496h52m55.238863614s) [noise=true , address=145.79.227.29:4942 , piece_id=P3NEFDCHGNYCHKNREO3W24UT3Y2JYDD7DRITY6VIHOCZHTQJ6UPQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.368647229 (496h52m55.239730275s) [noise=true , address=77.90.31.11:4554 , piece_id=NMGYD2NEH7JO5RYM2QK5J4SNMKVY4DULAWFI4JKRJACRH5A3N7CQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.368743052 (496h52m55.239826087s) [noise=true , address=145.79.227.42:4121 , piece_id=OZVKKT3LHSXVT3QYCPEOB4R55S3XKXGWOQCS5VLNFUKSHDDY62HQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.369129478 (496h52m55.240212523s) [noise=true , address=145.79.227.48:4301 , piece_id=EARSRIMER6XVHC7XHZ7IENWTZWGB4GKPVGRT7UIJEZV2DG3YG6ZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.370169819 (496h52m55.241252864s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.370629143 (496h52m55.241712179s) [noise=true , address=145.79.227.52:4445 , piece_id=2EGLPGSSAEW3JZEV4VRAXUEKC5KPWLMZQM54W3YR2XEJ5GCJM3WA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.371364123 (496h52m55.242447158s) [noise=true , address=145.79.227.33:4871 , piece_id=FK73ULAL6LXSYR3XKIENPGLDLQIRM62J7QEMY6ZOGNIRKP3UVSTQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.371776829 (496h52m55.242859874s) [noise=true , address=145.79.227.41:4087 , piece_id=N5LKUFBGUQZYQBPDLY5LVQ5BEMBNEJ2YY4VRJHUMUJ4CT67M5JLA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.372642968 (496h52m55.243726013s) [noise=true , address=45.91.116.11:4119 , piece_id=XH3DOGXGF4WUOQ5GZDTDSPV3ZLZ2A5GIGCZ4HM3LUVLYJP4KT57A, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.373850708 (496h52m55.244933753s) [noise=true , address=145.79.227.16:4909 , piece_id=3RGSW2XXGBZCFJM2AVCIIH7PVRTL4NQ4SGRL6Q2Z4WDK72RXFNOQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.374568565 (496h52m55.24565161s) [noise=true , address=145.79.227.6:4258 , piece_id=UEHDWUH3EA4AXU66HQ2MVYT6S7PX5OPJICLDGCOJHV7SQQ4HGOGA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.375882367 (496h52m55.246965402s) [noise=true , address=45.91.116.10:4082 , piece_id=JHC2BTBQMFIU6NKGDI2X5KB3EYSSIRDKORV46SD7U2O2YKHRWILA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:57.092354651 (496h52m56.963437686s) [upload=436109]: Next returned nil segment Dec 26 08:21:57.092370131 (496h52m56.963453167s) [upload=436109]: Waiting for error group managing segments... Dec 26 08:21:57.377216509 (496h52m57.248299555s) [noise=true , address=145.79.227.48:4301 , piece_id=EARSRIMER6XVHC7XHZ7IENWTZWGB4GKPVGRT7UIJEZV2DG3YG6ZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.381681541 (496h52m57.252764586s) [noise=true , address=145.79.227.52:4445 , piece_id=2EGLPGSSAEW3JZEV4VRAXUEKC5KPWLMZQM54W3YR2XEJ5GCJM3WA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.385419328 (496h52m57.256502373s) [noise=true , address=145.79.227.16:4909 , piece_id=3RGSW2XXGBZCFJM2AVCIIH7PVRTL4NQ4SGRL6Q2Z4WDK72RXFNOQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.385481707 (496h52m57.256564752s) [noise=true , address=145.79.227.29:4942 , piece_id=P3NEFDCHGNYCHKNREO3W24UT3Y2JYDD7DRITY6VIHOCZHTQJ6UPQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.392282417 (496h52m57.263365472s) [noise=true , address=145.79.227.42:4121 , piece_id=OZVKKT3LHSXVT3QYCPEOB4R55S3XKXGWOQCS5VLNFUKSHDDY62HQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.392560887 (496h52m57.263643922s) [noise=true , address=145.79.227.41:4087 , piece_id=N5LKUFBGUQZYQBPDLY5LVQ5BEMBNEJ2YY4VRJHUMUJ4CT67M5JLA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.403916531 (496h52m57.274999576s) [noise=true , address=145.79.227.7:4293 , piece_id=7HWLWI2VDKBNNU4PYZKKU2MI7HV7SEYI3DM3J3AM3KNFGDCTSSZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.405504594 (496h52m57.276587629s) [noise=true , address=145.79.227.46:4602 , piece_id=AXXAEY2OS66YPVQGRZWVSDRPRW5YZS4FUX2P3HDLDOLPSZ4ZLDWQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.406169921 (496h52m57.277252956s) [noise=true , address=145.79.227.15:4876 , piece_id=SROQM3MC4KIJWOZFQ6QHBEUR7M7OTPK7LPIVL4X537LOPVDMYDRQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.411074500 (496h52m57.282157545s) [noise=true , address=145.79.227.6:4258 , piece_id=UEHDWUH3EA4AXU66HQ2MVYT6S7PX5OPJICLDGCOJHV7SQQ4HGOGA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.500676197 (496h52m57.371759242s) [noise=true , address=145.79.227.33:4871 , piece_id=FK73ULAL6LXSYR3XKIENPGLDLQIRM62J7QEMY6ZOGNIRKP3UVSTQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.743736989 (496h52m57.614820024s) [noise=true , address=45.91.116.11:4119 , piece_id=XH3DOGXGF4WUOQ5GZDTDSPV3ZLZ2A5GIGCZ4HM3LUVLYJP4KT57A, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.921737773 (496h52m57.792820808s) [noise=true , address=77.90.31.11:4554 , piece_id=NMGYD2NEH7JO5RYM2QK5J4SNMKVY4DULAWFI4JKRJACRH5A3N7CQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 08:21:58.076627130 (496h52m57.947710165s) [noise=true , address=45.91.116.10:4082 , piece_id=JHC2BTBQMFIU6NKGDI2X5KB3EYSSIRDKORV46SD7U2O2YKHRWILA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:58.076647839 (496h52m57.947730884s) [seg_pos={0 0}, upload=436109]: Segment reached optimal threshold of 13 pieces. Dec 26 08:21:58.076946869 (496h52m57.948029904s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 08:21:58.076954663 (496h52m57.948037699s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Long tail cancellation due to optimal threshold Dec 26 08:21:58.077160215 (496h52m57.94824326s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:53900->45.91.116.13:4155: use of closed network connection; conn closed: op Dec 26 08:21:58.077170945 (496h52m57.94825398s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Long tail cancellation due to optimal threshold Dec 26 08:21:58.077194941 (496h52m57.948277986s) [seg_pos={0 0}, upload=436109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:21:58.077334998 (496h52m57.948418043s) [upload=436109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:21:58.077342832 (496h52m57.948425868s) [upload=436109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:21:58.363192762 (496h52m58.234275797s) [upload=436109]: Done uploading segments... Dec 26 08:21:58.363243778 (496h52m58.234326813s) [upload=436109]: Upload finished. err: <nil> Duration: 6.017194267s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.287548741s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:22:53 UTC) orbiter warning: context deadline exceeded Duration: 6.28757507s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.625151517s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:22:44 UTC) orbiter warning: context deadline exceeded Duration: 3.62517884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.....X.....XXX.X......................X..X.XX.X.X..X..X.XXX] duration: 30.003895039s trace: https://jaeger.dev.storj.io/trace/327144d892a2e847 FAIL (2025-12-26 08:22:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:21:55.122904853 (496h52m54.993987898s) [upload=436109]: Starting upload Dec 26 08:21:55.122979596 (496h52m54.994062641s) [upload=436109]: Done starting upload Dec 26 08:21:55.123079816 (496h52m54.994162851s) [upload=436109]: Uploading segments... Dec 26 08:21:55.123225254 (496h52m54.994308289s) [upload=436109]: Got next segment. Inline: false Dec 26 08:21:55.123288744 (496h52m54.994371789s) [upload=436109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:21:55.123295397 (496h52m54.994378442s) [upload=436109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:21:55.366353472 (496h52m55.237436507s) [seg_pos={0 0}, upload=436109]: Begin upload segment... Dec 26 08:21:55.366413947 (496h52m55.237496982s) [seg_pos={0 0}, upload=436109]: Done begin upload segment. Dec 26 08:21:55.366479612 (496h52m55.237562657s) [noise=true , address=145.79.227.46:4602 , piece_id=AXXAEY2OS66YPVQGRZWVSDRPRW5YZS4FUX2P3HDLDOLPSZ4ZLDWQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.366490302 (496h52m55.237573337s) [noise=true , address=145.79.227.15:4876 , piece_id=SROQM3MC4KIJWOZFQ6QHBEUR7M7OTPK7LPIVL4X537LOPVDMYDRQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.366558953 (496h52m55.237641998s) [noise=true , address=145.79.227.7:4293 , piece_id=7HWLWI2VDKBNNU4PYZKKU2MI7HV7SEYI3DM3J3AM3KNFGDCTSSZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.367158616 (496h52m55.238241681s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.367780569 (496h52m55.238863614s) [noise=true , address=145.79.227.29:4942 , piece_id=P3NEFDCHGNYCHKNREO3W24UT3Y2JYDD7DRITY6VIHOCZHTQJ6UPQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.368647229 (496h52m55.239730275s) [noise=true , address=77.90.31.11:4554 , piece_id=NMGYD2NEH7JO5RYM2QK5J4SNMKVY4DULAWFI4JKRJACRH5A3N7CQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.368743052 (496h52m55.239826087s) [noise=true , address=145.79.227.42:4121 , piece_id=OZVKKT3LHSXVT3QYCPEOB4R55S3XKXGWOQCS5VLNFUKSHDDY62HQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.369129478 (496h52m55.240212523s) [noise=true , address=145.79.227.48:4301 , piece_id=EARSRIMER6XVHC7XHZ7IENWTZWGB4GKPVGRT7UIJEZV2DG3YG6ZA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.370169819 (496h52m55.241252864s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.370629143 (496h52m55.241712179s) [noise=true , address=145.79.227.52:4445 , piece_id=2EGLPGSSAEW3JZEV4VRAXUEKC5KPWLMZQM54W3YR2XEJ5GCJM3WA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.371364123 (496h52m55.242447158s) [noise=true , address=145.79.227.33:4871 , piece_id=FK73ULAL6LXSYR3XKIENPGLDLQIRM62J7QEMY6ZOGNIRKP3UVSTQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.371776829 (496h52m55.242859874s) [noise=true , address=145.79.227.41:4087 , piece_id=N5LKUFBGUQZYQBPDLY5LVQ5BEMBNEJ2YY4VRJHUMUJ4CT67M5JLA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.372642968 (496h52m55.243726013s) [noise=true , address=45.91.116.11:4119 , piece_id=XH3DOGXGF4WUOQ5GZDTDSPV3ZLZ2A5GIGCZ4HM3LUVLYJP4KT57A, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.373850708 (496h52m55.244933753s) [noise=true , address=145.79.227.16:4909 , piece_id=3RGSW2XXGBZCFJM2AVCIIH7PVRTL4NQ4SGRL6Q2Z4WDK72RXFNOQ, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.374568565 (496h52m55.24565161s) [noise=true , address=145.79.227.6:4258 , piece_id=UEHDWUH3EA4AXU66HQ2MVYT6S7PX5OPJICLDGCOJHV7SQQ4HGOGA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:55.375882367 (496h52m55.246965402s) [noise=true , address=45.91.116.10:4082 , piece_id=JHC2BTBQMFIU6NKGDI2X5KB3EYSSIRDKORV46SD7U2O2YKHRWILA, seg_pos={0 0}, upload=436109]: Uploading piece... Dec 26 08:21:57.092354651 (496h52m56.963437686s) [upload=436109]: Next returned nil segment Dec 26 08:21:57.092370131 (496h52m56.963453167s) [upload=436109]: Waiting for error group managing segments... Dec 26 08:21:57.377216509 (496h52m57.248299555s) [noise=true , address=145.79.227.48:4301 , piece_id=EARSRIMER6XVHC7XHZ7IENWTZWGB4GKPVGRT7UIJEZV2DG3YG6ZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.381681541 (496h52m57.252764586s) [noise=true , address=145.79.227.52:4445 , piece_id=2EGLPGSSAEW3JZEV4VRAXUEKC5KPWLMZQM54W3YR2XEJ5GCJM3WA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.385419328 (496h52m57.256502373s) [noise=true , address=145.79.227.16:4909 , piece_id=3RGSW2XXGBZCFJM2AVCIIH7PVRTL4NQ4SGRL6Q2Z4WDK72RXFNOQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.385481707 (496h52m57.256564752s) [noise=true , address=145.79.227.29:4942 , piece_id=P3NEFDCHGNYCHKNREO3W24UT3Y2JYDD7DRITY6VIHOCZHTQJ6UPQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.392282417 (496h52m57.263365472s) [noise=true , address=145.79.227.42:4121 , piece_id=OZVKKT3LHSXVT3QYCPEOB4R55S3XKXGWOQCS5VLNFUKSHDDY62HQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.392560887 (496h52m57.263643922s) [noise=true , address=145.79.227.41:4087 , piece_id=N5LKUFBGUQZYQBPDLY5LVQ5BEMBNEJ2YY4VRJHUMUJ4CT67M5JLA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.403916531 (496h52m57.274999576s) [noise=true , address=145.79.227.7:4293 , piece_id=7HWLWI2VDKBNNU4PYZKKU2MI7HV7SEYI3DM3J3AM3KNFGDCTSSZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.405504594 (496h52m57.276587629s) [noise=true , address=145.79.227.46:4602 , piece_id=AXXAEY2OS66YPVQGRZWVSDRPRW5YZS4FUX2P3HDLDOLPSZ4ZLDWQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.406169921 (496h52m57.277252956s) [noise=true , address=145.79.227.15:4876 , piece_id=SROQM3MC4KIJWOZFQ6QHBEUR7M7OTPK7LPIVL4X537LOPVDMYDRQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.411074500 (496h52m57.282157545s) [noise=true , address=145.79.227.6:4258 , piece_id=UEHDWUH3EA4AXU66HQ2MVYT6S7PX5OPJICLDGCOJHV7SQQ4HGOGA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.500676197 (496h52m57.371759242s) [noise=true , address=145.79.227.33:4871 , piece_id=FK73ULAL6LXSYR3XKIENPGLDLQIRM62J7QEMY6ZOGNIRKP3UVSTQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.743736989 (496h52m57.614820024s) [noise=true , address=45.91.116.11:4119 , piece_id=XH3DOGXGF4WUOQ5GZDTDSPV3ZLZ2A5GIGCZ4HM3LUVLYJP4KT57A, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:57.921737773 (496h52m57.792820808s) [noise=true , address=77.90.31.11:4554 , piece_id=NMGYD2NEH7JO5RYM2QK5J4SNMKVY4DULAWFI4JKRJACRH5A3N7CQ, seg_pos={0 0}, upload=436109]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 08:21:58.076627130 (496h52m57.947710165s) [noise=true , address=45.91.116.10:4082 , piece_id=JHC2BTBQMFIU6NKGDI2X5KB3EYSSIRDKORV46SD7U2O2YKHRWILA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: <nil> Dec 26 08:21:58.076647839 (496h52m57.947730884s) [seg_pos={0 0}, upload=436109]: Segment reached optimal threshold of 13 pieces. Dec 26 08:21:58.076946869 (496h52m57.948029904s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 08:21:58.076954663 (496h52m57.948037699s) [noise=true , address=45.91.116.7:4010 , piece_id=TSKEHLIXNBZRL76XBIASVUSMXNSN4OPBRAZL5QUB253XQHBUJFKA, seg_pos={0 0}, upload=436109]: Long tail cancellation due to optimal threshold Dec 26 08:21:58.077160215 (496h52m57.94824326s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:53900->45.91.116.13:4155: use of closed network connection; conn closed: op Dec 26 08:21:58.077170945 (496h52m57.94825398s) [noise=true , address=45.91.116.13:4155 , piece_id=6BNGNVFFWPN3JRBYSTOYHSCQEMSLRRGLI22I2L2SA6AEJR6FBOZA, seg_pos={0 0}, upload=436109]: Long tail cancellation due to optimal threshold Dec 26 08:21:58.077194941 (496h52m57.948277986s) [seg_pos={0 0}, upload=436109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:21:58.077334998 (496h52m57.948418043s) [upload=436109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:21:58.077342832 (496h52m57.948425868s) [upload=436109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:21:58.363192762 (496h52m58.234275797s) [upload=436109]: Done uploading segments... Dec 26 08:21:58.363243778 (496h52m58.234326813s) [upload=436109]: Upload finished. err: <nil> Duration: 30.003911811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.....X.....XXX.X......................X..X.XX.X.X..X..X.XX] duration: 30.002785816s trace: https://jaeger.dev.storj.io/trace/3d764d12dde29a79 FAIL (2025-12-26 08:21:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 36 (496h52m24.777677081s) [upload=436101]: Done starting upload Dec 26 08:21:24.906717702 (496h52m24.777800747s) [upload=436101]: Uploading segments... Dec 26 08:21:24.906780852 (496h52m24.777863887s) [upload=436101]: Got next segment. Inline: false Dec 26 08:21:24.906806430 (496h52m24.777889465s) [upload=436101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:21:24.906810739 (496h52m24.777893784s) [upload=436101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:21:25.149964356 (496h52m25.021047391s) [seg_pos={0 0}, upload=436101]: Begin upload segment... Dec 26 08:21:25.150012909 (496h52m25.021095954s) [seg_pos={0 0}, upload=436101]: Done begin upload segment. Dec 26 08:21:25.150039800 (496h52m25.021122835s) [noise=true , address=77.90.31.10:4518 , piece_id=R42Z4UUCBY4ZW5NYK7ZGLO4RNRHCPQEUOY2AFNAGIOO3XDEHSSZA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.150132065 (496h52m25.0212151s) [noise=true , address=145.79.227.24:4581 , piece_id=7BFS646IPGEG7NP52X34ZJDRJFX7DWN47SDAJUQCBJJ6VZI7T5XA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.150868868 (496h52m25.021951903s) [noise=true , address=145.79.227.38:4978 , piece_id=3VYVC6AS2WZUUO6BOEIQEQYNVEHDE4YHRLZ36RGCT24XF4E32CHA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.151248962 (496h52m25.022331997s) [noise=true , address=77.90.31.14:4662 , piece_id=KNPWCWJNCNBXZ67YDUC4DYOXS2LBZENOMCXGSQB2C5LNNPLEX5IA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.151589791 (496h52m25.022672826s) [noise=true , address=145.79.227.17:4971 , piece_id=J4R52JKLPGZWKHRFJMKII6VTUVX4WCT57Q3A3PJOZMJ4NXODJZOQ, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.151989954 (496h52m25.023072999s) [noise=true , address=45.91.116.5:4300 , piece_id=VNHLBCDSLGZUBKP5BIMKK5O573HSTGDO4YZ56YAB76HDH3CTZTOA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.152324931 (496h52m25.023407966s) [noise=true , address=145.79.227.33:4871 , piece_id=4WLDQBDZGLWVYN2JQU3XFPR3GGOZESMUUVXIMNB5VFKENYLND6PQ, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.152637206 (496h52m25.023720241s) [noise=true , address=45.91.116.12:4226 , piece_id=YZHGFHGHPQLEP2B5RTT3CV255MJ7Z22Q676E6DN75SMUCPZ34NLQ, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.153018001 (496h52m25.024101046s) [noise=true , address=77.90.31.13:4627 , piece_id=IUI4Y5DYUQC3QGLP3PKGASVAVPGXCX4GXULRH2ZYGLZIMVO3AFDA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.153076292 (496h52m25.024159327s) [noise=true , address=145.79.227.29:4942 , piece_id=Q3XX3CYL4SNNA4TJI6TWDRRAUXLM6RLDEGAHOR3YEAR3TPVSO4NQ, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.153320497 (496h52m25.024403532s) [noise=true , address=145.79.227.5:4222 , piece_id=HSFDCRTUA5TTL55AECSSXZERXY6ZHEUQPBSHBBCISA4F6LB6I4ZA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.153606682 (496h52m25.024689727s) [noise=true , address=45.91.116.11:4117 , piece_id=4BNIMLA25ONO5AD3DF3JQ3JMU7RA33X6YKJ5C35JBVBCYVJWPWAA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.153925489 (496h52m25.025008524s) [noise=true , address=145.79.227.11:4730 , piece_id=44USALHNWJKV4F7XI7SVEED72OVZ52FQO54X3GNTQ3INEHTZ5DBA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.154139105 (496h52m25.02522214s) [noise=true , address=45.91.116.9:4262 , piece_id=B6CNW7X2BZE4U576KUOAJEJVMDGZFNVXA6BXREI23J5NQ4XUENKA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.154784384 (496h52m25.02586742s) [noise=true , address=145.79.227.45:4230 , piece_id=HUQ5M4Y55MVVUQDWNX4FJZMIDWWPVJIIW3N4QNNGQ5NBSUJMAPKA, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:25.156332362 (496h52m25.027415397s) [noise=true , address=77.90.31.17:4770 , piece_id=LMWLCY5NEWUE3ZQOG4U3W5YNUPMOWRLK2OJA7PMBN3TYWZPBJ2NQ, seg_pos={0 0}, upload=436101]: Uploading piece... Dec 26 08:21:26.869051230 (496h52m26.740134275s) [upload=436101]: Next returned nil segment Dec 26 08:21:26.869065909 (496h52m26.740148954s) [upload=436101]: Waiting for error group managing segments... Dec 26 08:21:27.158202795 (496h52m27.02928584s) [noise=true , address=145.79.227.24:4581 , piece_id=7BFS646IPGEG7NP52X34ZJDRJFX7DWN47SDAJUQCBJJ6VZI7T5XA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.161586689 (496h52m27.032669724s) [noise=true , address=145.79.227.38:4978 , piece_id=3VYVC6AS2WZUUO6BOEIQEQYNVEHDE4YHRLZ36RGCT24XF4E32CHA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.161839230 (496h52m27.032922265s) [noise=true , address=145.79.227.5:4222 , piece_id=HSFDCRTUA5TTL55AECSSXZERXY6ZHEUQPBSHBBCISA4F6LB6I4ZA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.162390009 (496h52m27.033473044s) [noise=true , address=145.79.227.17:4971 , piece_id=J4R52JKLPGZWKHRFJMKII6VTUVX4WCT57Q3A3PJOZMJ4NXODJZOQ, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.168990879 (496h52m27.040073914s) [noise=true , address=145.79.227.45:4230 , piece_id=HUQ5M4Y55MVVUQDWNX4FJZMIDWWPVJIIW3N4QNNGQ5NBSUJMAPKA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.178631747 (496h52m27.049714782s) [noise=true , address=145.79.227.11:4730 , piece_id=44USALHNWJKV4F7XI7SVEED72OVZ52FQO54X3GNTQ3INEHTZ5DBA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.179462208 (496h52m27.050545253s) [noise=true , address=145.79.227.29:4942 , piece_id=Q3XX3CYL4SNNA4TJI6TWDRRAUXLM6RLDEGAHOR3YEAR3TPVSO4NQ, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.261415100 (496h52m27.132498135s) [noise=true , address=145.79.227.33:4871 , piece_id=4WLDQBDZGLWVYN2JQU3XFPR3GGOZESMUUVXIMNB5VFKENYLND6PQ, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.339289657 (496h52m27.210372692s) [noise=true , address=77.90.31.10:4518 , piece_id=R42Z4UUCBY4ZW5NYK7ZGLO4RNRHCPQEUOY2AFNAGIOO3XDEHSSZA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.431007654 (496h52m27.302090689s) [noise=true , address=45.91.116.5:4300 , piece_id=VNHLBCDSLGZUBKP5BIMKK5O573HSTGDO4YZ56YAB76HDH3CTZTOA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.537784015 (496h52m27.40886708s) [noise=true , address=45.91.116.12:4226 , piece_id=YZHGFHGHPQLEP2B5RTT3CV255MJ7Z22Q676E6DN75SMUCPZ34NLQ, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.620052419 (496h52m27.491135464s) [noise=true , address=77.90.31.13:4627 , piece_id=IUI4Y5DYUQC3QGLP3PKGASVAVPGXCX4GXULRH2ZYGLZIMVO3AFDA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.724520154 (496h52m27.595603189s) [noise=true , address=45.91.116.11:4117 , piece_id=4BNIMLA25ONO5AD3DF3JQ3JMU7RA33X6YKJ5C35JBVBCYVJWPWAA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: <nil> Dec 26 08:21:27.724543057 (496h52m27.595626092s) [seg_pos={0 0}, upload=436101]: Segment reached optimal threshold of 13 pieces. Dec 26 08:21:27.724732418 (496h52m27.595815453s) [noise=true , address=77.90.31.17:4770 , piece_id=LMWLCY5NEWUE3ZQOG4U3W5YNUPMOWRLK2OJA7PMBN3TYWZPBJ2NQ, seg_pos={0 0}, upload=436101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:46462->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:46462->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46462->77.90.31.17:4770: use of closed network connection Dec 26 08:21:27.724738650 (496h52m27.595821685s) [noise=true , address=77.90.31.17:4770 , piece_id=LMWLCY5NEWUE3ZQOG4U3W5YNUPMOWRLK2OJA7PMBN3TYWZPBJ2NQ, seg_pos={0 0}, upload=436101]: Long tail cancellation due to optimal threshold Dec 26 08:21:27.724915907 (496h52m27.595998942s) [noise=true , address=77.90.31.14:4662 , piece_id=KNPWCWJNCNBXZ67YDUC4DYOXS2LBZENOMCXGSQB2C5LNNPLEX5IA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 08:21:27.724936857 (496h52m27.596019892s) [noise=true , address=77.90.31.14:4662 , piece_id=KNPWCWJNCNBXZ67YDUC4DYOXS2LBZENOMCXGSQB2C5LNNPLEX5IA, seg_pos={0 0}, upload=436101]: Long tail cancellation due to optimal threshold Dec 26 08:21:27.725086051 (496h52m27.596169086s) [noise=true , address=45.91.116.9:4262 , piece_id=B6CNW7X2BZE4U576KUOAJEJVMDGZFNVXA6BXREI23J5NQ4XUENKA, seg_pos={0 0}, upload=436101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: read tcp 188.166.225.170:47196->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:21:27.725093445 (496h52m27.59617648s) [noise=true , address=45.91.116.9:4262 , piece_id=B6CNW7X2BZE4U576KUOAJEJVMDGZFNVXA6BXREI23J5NQ4XUENKA, seg_pos={0 0}, upload=436101]: Long tail cancellation due to optimal threshold Dec 26 08:21:27.725109276 (496h52m27.596192311s) [seg_pos={0 0}, upload=436101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:21:27.725173218 (496h52m27.596256253s) [upload=436101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:21:27.725177987 (496h52m27.596261022s) [upload=436101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:21:28.008541691 (496h52m27.879624726s) [upload=436101]: Done uploading segments... Dec 26 08:21:28.008560706 (496h52m27.879643742s) [upload=436101]: Upload finished. err: <nil> Duration: 30.002807288s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.119078081s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:21:52 UTC) orbiter warning: context deadline exceeded Duration: 6.119104273s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.402437299s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:21:35 UTC) orbiter warning: context deadline exceeded Duration: 3.402462348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.510422301s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:21:11 UTC) orbiter warning: context deadline exceeded Duration: 3.510472236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.....X.....XXX.X......................X..X.XX.X.X..X..X.X] duration: 30.003316047s trace: https://jaeger.dev.storj.io/trace/5d16773a10cd11dc FAIL (2025-12-26 08:20:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: FCVAKU5JGLQW5KX3FRRAAUKECBLIGFNDKHBQX42HGHTEZMIMHYMQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 36C7JQQXXZOL347SEVMZHBKN7XYLMM6PQWS2DZM7C2AXUTC4YR7A) context deadline exceeded ======LOGS====== Dec 26 08:20:24.477526070 (496h51m24.348609106s) [upload=436090]: Starting upload Dec 26 08:20:24.477580424 (496h51m24.348663459s) [upload=436090]: Done starting upload Dec 26 08:20:24.477724268 (496h51m24.348807323s) [upload=436090]: Uploading segments... Dec 26 08:20:24.477837203 (496h51m24.348920238s) [upload=436090]: Got next segment. Inline: false Dec 26 08:20:24.477873763 (496h51m24.348956808s) [upload=436090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:20:24.477878451 (496h51m24.348961486s) [upload=436090]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:20:24.718949523 (496h51m24.590032558s) [seg_pos={0 0}, upload=436090]: Begin upload segment... Dec 26 08:20:24.718998918 (496h51m24.590081953s) [seg_pos={0 0}, upload=436090]: Done begin upload segment. Dec 26 08:20:24.719028604 (496h51m24.590111639s) [noise=true , address=77.90.31.17:4770 , piece_id=JQM2R2BSGFL74BCMCVDP3UNLOYNWA7FECBKKCR7MCUG2M7RH6VGQ, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.719045777 (496h51m24.590128812s) [noise=true , address=145.79.227.5:4221 , piece_id=3ANEKEDWQQRI5XPDWNRXJQPDZ6IK5R36NGXGTUEGAYPU6CAHDG3A, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.719504360 (496h51m24.590587395s) [noise=true , address=45.91.116.11:4117 , piece_id=IIBGIDLNX5AWX6QTYPIDWIZRPS4IJJFEZ5TF7UZPNUIFWSCRHDDQ, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.719853474 (496h51m24.590936509s) [noise=true , address=45.91.116.9:4262 , piece_id=BYNW4E7LXHIW7NWMQW2ZQPJQ2K35NMA2XKK22ECJUJFEYL332P5A, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.719899994 (496h51m24.590983029s) [noise=true , address=145.79.227.40:4049 , piece_id=CFBPPHSBF2VKLJZJGF5I75POIDL7SDVGVZQWNVOOH5LEGR75RRYA, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.720211998 (496h51m24.591295033s) [noise=true , address=145.79.227.18:4006 , piece_id=4J4CVLH77B6N7YOIGVX3EFOV44E57BA6JBZW6YLDAWE6NZKFC2UQ, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.720291689 (496h51m24.591374724s) [noise=true , address=45.91.116.12:4226 , piece_id=36C7JQQXXZOL347SEVMZHBKN7XYLMM6PQWS2DZM7C2AXUTC4YR7A, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.720483485 (496h51m24.59156652s) [noise=true , address=145.79.227.42:4123 , piece_id=TWMC7EQG6APEWYMEW7HZD6QZESBEEM4BQAY6XTVEY6GHUORRQ3UA, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.720562645 (496h51m24.59164568s) [noise=true , address=45.91.116.5:4300 , piece_id=FCVAKU5JGLQW5KX3FRRAAUKECBLIGFNDKHBQX42HGHTEZMIMHYMQ, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.720753268 (496h51m24.591836303s) [noise=true , address=77.90.31.14:4663 , piece_id=3FNS4XYJZE7V4DFLWHKVE3LA7BPUE42T6XQS2Z4XIYLSTLWHAGPA, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.720832039 (496h51m24.591915084s) [noise=true , address=45.91.116.6:4974 , piece_id=G4J4XUBQDYSLA6HDCZ2ZGRXFLDKOWE5VSDIUDT7X2L7OKPWZNBWA, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.721090290 (496h51m24.592173335s) [noise=true , address=77.90.31.10:4518 , piece_id=HDTVFZ3YLCPUVSXXPE34CYXZBTH6XZYBFLX5GCE2XBQQYSEPDRMA, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.721129755 (496h51m24.5922128s) [noise=true , address=145.79.227.45:4230 , piece_id=3U3EFUD6QZLXRURCCYMKBAKRYOTGYSQFLMNXCONSGAEKLNTU2OZA, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.721419356 (496h51m24.592502391s) [noise=true , address=145.79.227.39:4014 , piece_id=RPPU27MUU7Y2QYBUOSXVLTMKCDTOP52YEZAPHAUD3CJR3NZEXOTQ, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.721428274 (496h51m24.592511309s) [noise=true , address=145.79.227.33:4871 , piece_id=XINBQEWEZ3HO4OBXVX74T2EMCVBQL5YEAK5JBS6UVVRBJBCCSS5A, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:24.721681796 (496h51m24.592764831s) [noise=true , address=145.79.227.36:4518 , piece_id=V374N2635QIHUOPP32ZQYDSRVPPTJLBUAKO5TT2CQEYGQK33HPCQ, seg_pos={0 0}, upload=436090]: Uploading piece... Dec 26 08:20:26.424608759 (496h51m26.295691804s) [upload=436090]: Next returned nil segment Dec 26 08:20:26.424639888 (496h51m26.295722923s) [upload=436090]: Waiting for error group managing segments... Dec 26 08:20:26.713013713 (496h51m26.584096758s) [noise=true , address=145.79.227.18:4006 , piece_id=4J4CVLH77B6N7YOIGVX3EFOV44E57BA6JBZW6YLDAWE6NZKFC2UQ, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.739334412 (496h51m26.610417447s) [noise=true , address=145.79.227.39:4014 , piece_id=RPPU27MUU7Y2QYBUOSXVLTMKCDTOP52YEZAPHAUD3CJR3NZEXOTQ, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.743984456 (496h51m26.615067492s) [noise=true , address=145.79.227.36:4518 , piece_id=V374N2635QIHUOPP32ZQYDSRVPPTJLBUAKO5TT2CQEYGQK33HPCQ, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.748512539 (496h51m26.619595574s) [noise=true , address=145.79.227.45:4230 , piece_id=3U3EFUD6QZLXRURCCYMKBAKRYOTGYSQFLMNXCONSGAEKLNTU2OZA, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.751907192 (496h51m26.622990227s) [noise=true , address=145.79.227.42:4123 , piece_id=TWMC7EQG6APEWYMEW7HZD6QZESBEEM4BQAY6XTVEY6GHUORRQ3UA, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.752194270 (496h51m26.623277305s) [noise=true , address=145.79.227.5:4221 , piece_id=3ANEKEDWQQRI5XPDWNRXJQPDZ6IK5R36NGXGTUEGAYPU6CAHDG3A, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.783289881 (496h51m26.654372916s) [noise=true , address=145.79.227.33:4871 , piece_id=XINBQEWEZ3HO4OBXVX74T2EMCVBQL5YEAK5JBS6UVVRBJBCCSS5A, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.887336164 (496h51m26.758419199s) [noise=true , address=77.90.31.10:4518 , piece_id=HDTVFZ3YLCPUVSXXPE34CYXZBTH6XZYBFLX5GCE2XBQQYSEPDRMA, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.949861372 (496h51m26.820944418s) [noise=true , address=77.90.31.14:4663 , piece_id=3FNS4XYJZE7V4DFLWHKVE3LA7BPUE42T6XQS2Z4XIYLSTLWHAGPA, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.963335810 (496h51m26.834418845s) [noise=true , address=45.91.116.12:4226 , piece_id=36C7JQQXXZOL347SEVMZHBKN7XYLMM6PQWS2DZM7C2AXUTC4YR7A, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.982409059 (496h51m26.853492104s) [noise=true , address=45.91.116.9:4262 , piece_id=BYNW4E7LXHIW7NWMQW2ZQPJQ2K35NMA2XKK22ECJUJFEYL332P5A, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:26.989880727 (496h51m26.860963762s) [noise=true , address=45.91.116.11:4117 , piece_id=IIBGIDLNX5AWX6QTYPIDWIZRPS4IJJFEZ5TF7UZPNUIFWSCRHDDQ, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:27.061585044 (496h51m26.932668089s) [noise=true , address=45.91.116.5:4300 , piece_id=FCVAKU5JGLQW5KX3FRRAAUKECBLIGFNDKHBQX42HGHTEZMIMHYMQ, seg_pos={0 0}, upload=436090]: Done uploading piece. err: <nil> Dec 26 08:20:27.061610443 (496h51m26.932693488s) [seg_pos={0 0}, upload=436090]: Segment reached optimal threshold of 13 pieces. Dec 26 08:20:27.061876950 (496h51m26.932960005s) [noise=true , address=145.79.227.40:4049 , piece_id=CFBPPHSBF2VKLJZJGF5I75POIDL7SDVGVZQWNVOOH5LEGR75RRYA, seg_pos={0 0}, upload=436090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 08:20:27.061886719 (496h51m26.932969764s) [noise=true , address=145.79.227.40:4049 , piece_id=CFBPPHSBF2VKLJZJGF5I75POIDL7SDVGVZQWNVOOH5LEGR75RRYA, seg_pos={0 0}, upload=436090]: Long tail cancellation due to optimal threshold Dec 26 08:20:27.062219151 (496h51m26.933302217s) [noise=true , address=45.91.116.6:4974 , piece_id=G4J4XUBQDYSLA6HDCZ2ZGRXFLDKOWE5VSDIUDT7X2L7OKPWZNBWA, seg_pos={0 0}, upload=436090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 08:20:27.062231125 (496h51m26.93331419s) [noise=true , address=45.91.116.6:4974 , piece_id=G4J4XUBQDYSLA6HDCZ2ZGRXFLDKOWE5VSDIUDT7X2L7OKPWZNBWA, seg_pos={0 0}, upload=436090]: Long tail cancellation due to optimal threshold Dec 26 08:20:27.062521528 (496h51m26.933604573s) [noise=true , address=77.90.31.17:4770 , piece_id=JQM2R2BSGFL74BCMCVDP3UNLOYNWA7FECBKKCR7MCUG2M7RH6VGQ, seg_pos={0 0}, upload=436090]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 08:20:27.062530574 (496h51m26.93361362s) [noise=true , address=77.90.31.17:4770 , piece_id=JQM2R2BSGFL74BCMCVDP3UNLOYNWA7FECBKKCR7MCUG2M7RH6VGQ, seg_pos={0 0}, upload=436090]: Long tail cancellation due to optimal threshold Dec 26 08:20:27.062552186 (496h51m26.933635241s) [seg_pos={0 0}, upload=436090]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:20:27.062649822 (496h51m26.933732867s) [upload=436090]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:20:27.062659831 (496h51m26.933742896s) [upload=436090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:20:27.353101432 (496h51m27.224184477s) [upload=436090]: Done uploading segments... Dec 26 08:20:27.353155946 (496h51m27.224238981s) [upload=436090]: Upload finished. err: <nil> Duration: 30.003342057s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.416224251s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:20:47 UTC) orbiter warning: context deadline exceeded Duration: 6.416254127s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.404891201s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:20:15 UTC) orbiter warning: context deadline exceeded Duration: 3.404922812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.....X.....XXX.X......................X..X.XX.X.X..X..X.] duration: 5.868151163s trace: https://jaeger.dev.storj.io/trace/458d0f6245f78b06 FAIL (2025-12-26 08:19:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA) context deadline exceeded ======LOGS====== Dec 26 08:18:52.134819319 (496h49m52.005902364s) [upload=436067]: Starting upload Dec 26 08:18:52.134906055 (496h49m52.0059891s) [upload=436067]: Done starting upload Dec 26 08:18:52.134979795 (496h49m52.00606283s) [upload=436067]: Uploading segments... Dec 26 08:18:52.135193111 (496h49m52.006276146s) [upload=436067]: Got next segment. Inline: false Dec 26 08:18:52.135222517 (496h49m52.006305552s) [upload=436067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:18:52.135227096 (496h49m52.006310131s) [upload=436067]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:18:52.382805990 (496h49m52.253889065s) [seg_pos={0 0}, upload=436067]: Begin upload segment... Dec 26 08:18:52.382860142 (496h49m52.253943187s) [seg_pos={0 0}, upload=436067]: Done begin upload segment. Dec 26 08:18:52.382897624 (496h49m52.253980669s) [noise=true , address=145.79.227.45:4230 , piece_id=2ML7SHQ7XJRI3R6F5AJ6SZ4HW5KF5MLA23SM5VVOGNQUV4VDL3NQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.382945435 (496h49m52.25402847s) [noise=true , address=145.79.227.17:4970 , piece_id=YN6VSEO4P2RXL3BWTHHZJKETAP3BBEA2THMDJPEHB62APJ6PD53A, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.383473030 (496h49m52.254556065s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.383903780 (496h49m52.254986815s) [noise=true , address=77.90.31.13:4629 , piece_id=MIZKIVIFAIWICTIOFYHPI5ABHNUQHWHDZNSWYPETKENGVSRGSPUQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.384356261 (496h49m52.255439306s) [noise=true , address=145.79.227.51:4409 , piece_id=WF6XZJG7KX7I2Z4RD4KIPDBAKUVUO7CFG5WX4LSK5SSC7JLZXRBA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.384773626 (496h49m52.255856671s) [noise=true , address=145.79.227.12:4767 , piece_id=KUAFPNTEJCOGQOSLR2U5GLJHKAYERYB4XIUQDRMYIDBTTXHOXHXQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.385218363 (496h49m52.256301398s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.385710881 (496h49m52.256793916s) [noise=true , address=145.79.227.42:4122 , piece_id=W24LQ3LDO6KVYR3P7KR3UY4IYRPI7G33LDK2XA3PYR6VNXYZGEMA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.386836825 (496h49m52.257919861s) [noise=true , address=145.79.227.41:4087 , piece_id=CK46LF4X6PZ3UM632FRNWVA65H7HQQGDK3RVFCQS3LSX2Y75HSHA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.387375611 (496h49m52.258458646s) [noise=true , address=77.90.31.16:4736 , piece_id=F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.388295623 (496h49m52.259378658s) [noise=true , address=145.79.227.37:4565 , piece_id=JU5JOBMNSUZ3BXJ7O5U6DW4FNCD4A5W7P2JLQGZU6RBEOGWCNLOQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.391014010 (496h49m52.262097045s) [noise=true , address=145.79.227.15:4876 , piece_id=MAOK3GSF5LIL4CMAWL5WBB4T36H345QJPE5DUNDCVGZKCFPNNEQQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.394846126 (496h49m52.265929172s) [noise=true , address=45.91.116.4:4939 , piece_id=UCG5MDXWS7B5PU2MRAGG7EYRSTG4KJ3SFLJZUZJTNN7NFSWBAIZQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.399866796 (496h49m52.270949842s) [noise=true , address=145.79.227.6:4258 , piece_id=PASXVXIG6XTYTPXYRPIB56C2C5DNDJ7H3EJ7PG7PEMTFXFAQDEDQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.401440884 (496h49m52.272523929s) [noise=true , address=145.79.227.38:4979 , piece_id=3XUAB6QG2KCGZH2OG3VHLGY6WPHKHFA5XXDDCDKNUDIDNYHY4KVA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.404991454 (496h49m52.27607451s) [noise=true , address=77.90.31.14:4664 , piece_id=WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:53.722345789 (496h49m53.593428834s) [noise=true , address=145.79.227.37:4565 , piece_id=JU5JOBMNSUZ3BXJ7O5U6DW4FNCD4A5W7P2JLQGZU6RBEOGWCNLOQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: storage node overloaded, request limit: 2000 Dec 26 08:18:54.122882432 (496h49m53.993965467s) [upload=436067]: Next returned nil segment Dec 26 08:18:54.122896939 (496h49m53.993979984s) [upload=436067]: Waiting for error group managing segments... Dec 26 08:18:54.396051093 (496h49m54.267134128s) [noise=true , address=145.79.227.45:4230 , piece_id=2ML7SHQ7XJRI3R6F5AJ6SZ4HW5KF5MLA23SM5VVOGNQUV4VDL3NQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.403055701 (496h49m54.274138736s) [noise=true , address=145.79.227.41:4087 , piece_id=CK46LF4X6PZ3UM632FRNWVA65H7HQQGDK3RVFCQS3LSX2Y75HSHA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.408596261 (496h49m54.279679306s) [noise=true , address=145.79.227.6:4258 , piece_id=PASXVXIG6XTYTPXYRPIB56C2C5DNDJ7H3EJ7PG7PEMTFXFAQDEDQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.409983382 (496h49m54.281066417s) [noise=true , address=145.79.227.38:4979 , piece_id=3XUAB6QG2KCGZH2OG3VHLGY6WPHKHFA5XXDDCDKNUDIDNYHY4KVA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.415846607 (496h49m54.286929643s) [noise=true , address=145.79.227.42:4122 , piece_id=W24LQ3LDO6KVYR3P7KR3UY4IYRPI7G33LDK2XA3PYR6VNXYZGEMA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.420956807 (496h49m54.292039852s) [noise=true , address=145.79.227.12:4767 , piece_id=KUAFPNTEJCOGQOSLR2U5GLJHKAYERYB4XIUQDRMYIDBTTXHOXHXQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.422031644 (496h49m54.293114679s) [noise=true , address=145.79.227.51:4409 , piece_id=WF6XZJG7KX7I2Z4RD4KIPDBAKUVUO7CFG5WX4LSK5SSC7JLZXRBA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.428515050 (496h49m54.299598085s) [noise=true , address=145.79.227.17:4970 , piece_id=YN6VSEO4P2RXL3BWTHHZJKETAP3BBEA2THMDJPEHB62APJ6PD53A, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.441698103 (496h49m54.312781148s) [noise=true , address=145.79.227.15:4876 , piece_id=MAOK3GSF5LIL4CMAWL5WBB4T36H345QJPE5DUNDCVGZKCFPNNEQQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.603821214 (496h49m54.474904249s) [noise=true , address=77.90.31.14:4664 , piece_id=WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.606761101 (496h49m54.477844146s) [noise=true , address=77.90.31.16:4736 , piece_id=F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.611521737 (496h49m54.482604772s) [noise=true , address=77.90.31.13:4629 , piece_id=MIZKIVIFAIWICTIOFYHPI5ABHNUQHWHDZNSWYPETKENGVSRGSPUQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.632211042 (496h49m54.503294077s) [noise=true , address=45.91.116.4:4939 , piece_id=UCG5MDXWS7B5PU2MRAGG7EYRSTG4KJ3SFLJZUZJTNN7NFSWBAIZQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.632233024 (496h49m54.50331606s) [seg_pos={0 0}, upload=436067]: Segment reached optimal threshold of 13 pieces. Dec 26 08:18:54.632510373 (496h49m54.503593418s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 08:18:54.632559345 (496h49m54.503642401s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Long tail cancellation due to optimal threshold Dec 26 08:18:54.632777912 (496h49m54.503860957s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 08:18:54.632792119 (496h49m54.503875154s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Long tail cancellation due to optimal threshold Dec 26 08:18:54.632815003 (496h49m54.503898048s) [seg_pos={0 0}, upload=436067]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:18:54.632934260 (496h49m54.504017295s) [upload=436067]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:18:54.632941122 (496h49m54.504024168s) [upload=436067]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:18:54.925285408 (496h49m54.796368443s) [upload=436067]: Done uploading segments... Dec 26 08:18:54.925304103 (496h49m54.796387148s) [upload=436067]: Upload finished. err: <nil> Duration: 5.868183523s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.98700468s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:19:47 UTC) orbiter warning: context deadline exceeded Duration: 5.98703097s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.459976716s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:19:39 UTC) orbiter warning: context deadline exceeded Duration: 3.460001453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...X.....X.....XXX.X......................X..X.XX.X.X..X..X] duration: 30.002538367s trace: https://jaeger.dev.storj.io/trace/458d0f6245f78b06 FAIL (2025-12-26 08:19:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA) context deadline exceeded ======LOGS====== Dec 26 08:18:52.134819319 (496h49m52.005902364s) [upload=436067]: Starting upload Dec 26 08:18:52.134906055 (496h49m52.0059891s) [upload=436067]: Done starting upload Dec 26 08:18:52.134979795 (496h49m52.00606283s) [upload=436067]: Uploading segments... Dec 26 08:18:52.135193111 (496h49m52.006276146s) [upload=436067]: Got next segment. Inline: false Dec 26 08:18:52.135222517 (496h49m52.006305552s) [upload=436067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:18:52.135227096 (496h49m52.006310131s) [upload=436067]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:18:52.382805990 (496h49m52.253889065s) [seg_pos={0 0}, upload=436067]: Begin upload segment... Dec 26 08:18:52.382860142 (496h49m52.253943187s) [seg_pos={0 0}, upload=436067]: Done begin upload segment. Dec 26 08:18:52.382897624 (496h49m52.253980669s) [noise=true , address=145.79.227.45:4230 , piece_id=2ML7SHQ7XJRI3R6F5AJ6SZ4HW5KF5MLA23SM5VVOGNQUV4VDL3NQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.382945435 (496h49m52.25402847s) [noise=true , address=145.79.227.17:4970 , piece_id=YN6VSEO4P2RXL3BWTHHZJKETAP3BBEA2THMDJPEHB62APJ6PD53A, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.383473030 (496h49m52.254556065s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.383903780 (496h49m52.254986815s) [noise=true , address=77.90.31.13:4629 , piece_id=MIZKIVIFAIWICTIOFYHPI5ABHNUQHWHDZNSWYPETKENGVSRGSPUQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.384356261 (496h49m52.255439306s) [noise=true , address=145.79.227.51:4409 , piece_id=WF6XZJG7KX7I2Z4RD4KIPDBAKUVUO7CFG5WX4LSK5SSC7JLZXRBA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.384773626 (496h49m52.255856671s) [noise=true , address=145.79.227.12:4767 , piece_id=KUAFPNTEJCOGQOSLR2U5GLJHKAYERYB4XIUQDRMYIDBTTXHOXHXQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.385218363 (496h49m52.256301398s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.385710881 (496h49m52.256793916s) [noise=true , address=145.79.227.42:4122 , piece_id=W24LQ3LDO6KVYR3P7KR3UY4IYRPI7G33LDK2XA3PYR6VNXYZGEMA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.386836825 (496h49m52.257919861s) [noise=true , address=145.79.227.41:4087 , piece_id=CK46LF4X6PZ3UM632FRNWVA65H7HQQGDK3RVFCQS3LSX2Y75HSHA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.387375611 (496h49m52.258458646s) [noise=true , address=77.90.31.16:4736 , piece_id=F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.388295623 (496h49m52.259378658s) [noise=true , address=145.79.227.37:4565 , piece_id=JU5JOBMNSUZ3BXJ7O5U6DW4FNCD4A5W7P2JLQGZU6RBEOGWCNLOQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.391014010 (496h49m52.262097045s) [noise=true , address=145.79.227.15:4876 , piece_id=MAOK3GSF5LIL4CMAWL5WBB4T36H345QJPE5DUNDCVGZKCFPNNEQQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.394846126 (496h49m52.265929172s) [noise=true , address=45.91.116.4:4939 , piece_id=UCG5MDXWS7B5PU2MRAGG7EYRSTG4KJ3SFLJZUZJTNN7NFSWBAIZQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.399866796 (496h49m52.270949842s) [noise=true , address=145.79.227.6:4258 , piece_id=PASXVXIG6XTYTPXYRPIB56C2C5DNDJ7H3EJ7PG7PEMTFXFAQDEDQ, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.401440884 (496h49m52.272523929s) [noise=true , address=145.79.227.38:4979 , piece_id=3XUAB6QG2KCGZH2OG3VHLGY6WPHKHFA5XXDDCDKNUDIDNYHY4KVA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:52.404991454 (496h49m52.27607451s) [noise=true , address=77.90.31.14:4664 , piece_id=WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA, seg_pos={0 0}, upload=436067]: Uploading piece... Dec 26 08:18:53.722345789 (496h49m53.593428834s) [noise=true , address=145.79.227.37:4565 , piece_id=JU5JOBMNSUZ3BXJ7O5U6DW4FNCD4A5W7P2JLQGZU6RBEOGWCNLOQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: storage node overloaded, request limit: 2000 Dec 26 08:18:54.122882432 (496h49m53.993965467s) [upload=436067]: Next returned nil segment Dec 26 08:18:54.122896939 (496h49m53.993979984s) [upload=436067]: Waiting for error group managing segments... Dec 26 08:18:54.396051093 (496h49m54.267134128s) [noise=true , address=145.79.227.45:4230 , piece_id=2ML7SHQ7XJRI3R6F5AJ6SZ4HW5KF5MLA23SM5VVOGNQUV4VDL3NQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.403055701 (496h49m54.274138736s) [noise=true , address=145.79.227.41:4087 , piece_id=CK46LF4X6PZ3UM632FRNWVA65H7HQQGDK3RVFCQS3LSX2Y75HSHA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.408596261 (496h49m54.279679306s) [noise=true , address=145.79.227.6:4258 , piece_id=PASXVXIG6XTYTPXYRPIB56C2C5DNDJ7H3EJ7PG7PEMTFXFAQDEDQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.409983382 (496h49m54.281066417s) [noise=true , address=145.79.227.38:4979 , piece_id=3XUAB6QG2KCGZH2OG3VHLGY6WPHKHFA5XXDDCDKNUDIDNYHY4KVA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.415846607 (496h49m54.286929643s) [noise=true , address=145.79.227.42:4122 , piece_id=W24LQ3LDO6KVYR3P7KR3UY4IYRPI7G33LDK2XA3PYR6VNXYZGEMA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.420956807 (496h49m54.292039852s) [noise=true , address=145.79.227.12:4767 , piece_id=KUAFPNTEJCOGQOSLR2U5GLJHKAYERYB4XIUQDRMYIDBTTXHOXHXQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.422031644 (496h49m54.293114679s) [noise=true , address=145.79.227.51:4409 , piece_id=WF6XZJG7KX7I2Z4RD4KIPDBAKUVUO7CFG5WX4LSK5SSC7JLZXRBA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.428515050 (496h49m54.299598085s) [noise=true , address=145.79.227.17:4970 , piece_id=YN6VSEO4P2RXL3BWTHHZJKETAP3BBEA2THMDJPEHB62APJ6PD53A, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.441698103 (496h49m54.312781148s) [noise=true , address=145.79.227.15:4876 , piece_id=MAOK3GSF5LIL4CMAWL5WBB4T36H345QJPE5DUNDCVGZKCFPNNEQQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.603821214 (496h49m54.474904249s) [noise=true , address=77.90.31.14:4664 , piece_id=WTN4JY76WAGP7BI6R3HGDC42ZU6CBJFHR7BZGJ2WY4UL6WKPBIQA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.606761101 (496h49m54.477844146s) [noise=true , address=77.90.31.16:4736 , piece_id=F55ROZ3JQQSGKBEXFQH6T2D4K3UKXUKJM4LF6DIGXPSCJ27KNP6A, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.611521737 (496h49m54.482604772s) [noise=true , address=77.90.31.13:4629 , piece_id=MIZKIVIFAIWICTIOFYHPI5ABHNUQHWHDZNSWYPETKENGVSRGSPUQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.632211042 (496h49m54.503294077s) [noise=true , address=45.91.116.4:4939 , piece_id=UCG5MDXWS7B5PU2MRAGG7EYRSTG4KJ3SFLJZUZJTNN7NFSWBAIZQ, seg_pos={0 0}, upload=436067]: Done uploading piece. err: <nil> Dec 26 08:18:54.632233024 (496h49m54.50331606s) [seg_pos={0 0}, upload=436067]: Segment reached optimal threshold of 13 pieces. Dec 26 08:18:54.632510373 (496h49m54.503593418s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 08:18:54.632559345 (496h49m54.503642401s) [noise=true , address=45.91.116.9:4261 , piece_id=TBKHI7XHGFPA2D6FAEXSHKT64554KX2TAB4UAYJQDTYP6M4Z3MFA, seg_pos={0 0}, upload=436067]: Long tail cancellation due to optimal threshold Dec 26 08:18:54.632777912 (496h49m54.503860957s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 08:18:54.632792119 (496h49m54.503875154s) [noise=true , address=45.91.116.14:4190 , piece_id=7K6TK7DI5AE3IZ456UWV3V4HQFY3DBJDYFMU7ZCKYCOECTUKPBQA, seg_pos={0 0}, upload=436067]: Long tail cancellation due to optimal threshold Dec 26 08:18:54.632815003 (496h49m54.503898048s) [seg_pos={0 0}, upload=436067]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:18:54.632934260 (496h49m54.504017295s) [upload=436067]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:18:54.632941122 (496h49m54.504024168s) [upload=436067]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:18:54.925285408 (496h49m54.796368443s) [upload=436067]: Done uploading segments... Dec 26 08:18:54.925304103 (496h49m54.796387148s) [upload=436067]: Upload finished. err: <nil> Duration: 30.002559457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.564987775s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:18:55 UTC) orbiter warning: context deadline exceeded Duration: 3.565012743s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.145924722s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:18:52 UTC) orbiter warning: context deadline exceeded Duration: 6.145951213s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.356809146s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:18:26 UTC) orbiter warning: context deadline exceeded Duration: 3.356834504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.970244585s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:18:24 UTC) orbiter warning: context deadline exceeded Duration: 5.970293818s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.936524471s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:18:03 UTC) orbiter warning: context deadline exceeded Duration: 6.936569246s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...X.....X.....XXX.X......................X..X.XX.X.X..X..] duration: 6.01618937s trace: https://jaeger.dev.storj.io/trace/44b8d37bbaf1cbd0 FAIL (2025-12-26 08:17:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA) context deadline exceeded ======LOGS====== 82 (496h47m3.620943537s) [upload=436031]: Uploading segments... Dec 26 08:16:03.750170222 (496h47m3.621253337s) [upload=436031]: Got next segment. Inline: false Dec 26 08:16:03.750217301 (496h47m3.621300336s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:16:03.750225136 (496h47m3.621308181s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:16:03.996028027 (496h47m3.867111082s) [seg_pos={0 0}, upload=436031]: Begin upload segment... Dec 26 08:16:03.996201377 (496h47m3.867284422s) [seg_pos={0 0}, upload=436031]: Done begin upload segment. Dec 26 08:16:03.996260529 (496h47m3.867343574s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996356201 (496h47m3.867439246s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996987703 (496h47m3.868070749s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.997416861 (496h47m3.868499906s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998071046 (496h47m3.869154091s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998678824 (496h47m3.869761869s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999124643 (496h47m3.870207688s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999669781 (496h47m3.870752826s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.000416904 (496h47m3.871499959s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002163089 (496h47m3.873246134s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002923166 (496h47m3.874006221s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.003684706 (496h47m3.874767771s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004377015 (496h47m3.87546007s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004494839 (496h47m3.875577884s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.005322004 (496h47m3.876405049s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.006274337 (496h47m3.877357372s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:05.709486098 (496h47m5.580569143s) [upload=436031]: Next returned nil segment Dec 26 08:16:05.709504011 (496h47m5.580587046s) [upload=436031]: Waiting for error group managing segments... Dec 26 08:16:05.997101825 (496h47m5.86818486s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.002826386 (496h47m5.873909431s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.003169179 (496h47m5.874252244s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.015164881 (496h47m5.886247916s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.017734023 (496h47m5.888817068s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.018932005 (496h47m5.89001504s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.020254483 (496h47m5.891337528s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.024683606 (496h47m5.895766651s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036334721 (496h47m5.907417756s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036433650 (496h47m5.907516695s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.251058020 (496h47m6.122141065s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.355066080 (496h47m6.226149115s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376711365 (496h47m6.2477944s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376731875 (496h47m6.24781491s) [seg_pos={0 0}, upload=436031]: Segment reached optimal threshold of 13 pieces. Dec 26 08:16:06.376924802 (496h47m6.248007837s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection Dec 26 08:16:06.376932617 (496h47m6.248015652s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.376963205 (496h47m6.24804625s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 08:16:06.376973816 (496h47m6.248056861s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377029190 (496h47m6.248112236s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection Dec 26 08:16:06.377035894 (496h47m6.248118929s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377050131 (496h47m6.248133166s) [seg_pos={0 0}, upload=436031]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:16:06.377121917 (496h47m6.248204952s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:16:06.377127738 (496h47m6.248210773s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:16:06.671006954 (496h47m6.542089989s) [upload=436031]: Done uploading segments... Dec 26 08:16:06.671028706 (496h47m6.542111741s) [upload=436031]: Upload finished. err: <nil> Duration: 6.016218046s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.883125384s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:17:41 UTC) orbiter warning: context deadline exceeded Duration: 5.883150892s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.461669739s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:17:34 UTC) orbiter warning: context deadline exceeded Duration: 3.461700296s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X...X.....X.....XXX.X......................X..X.XX.X.X..X.] duration: 6.072190278s trace: https://jaeger.dev.storj.io/trace/44b8d37bbaf1cbd0 FAIL (2025-12-26 08:17:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA) context deadline exceeded ======LOGS====== 82 (496h47m3.620943537s) [upload=436031]: Uploading segments... Dec 26 08:16:03.750170222 (496h47m3.621253337s) [upload=436031]: Got next segment. Inline: false Dec 26 08:16:03.750217301 (496h47m3.621300336s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:16:03.750225136 (496h47m3.621308181s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:16:03.996028027 (496h47m3.867111082s) [seg_pos={0 0}, upload=436031]: Begin upload segment... Dec 26 08:16:03.996201377 (496h47m3.867284422s) [seg_pos={0 0}, upload=436031]: Done begin upload segment. Dec 26 08:16:03.996260529 (496h47m3.867343574s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996356201 (496h47m3.867439246s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996987703 (496h47m3.868070749s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.997416861 (496h47m3.868499906s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998071046 (496h47m3.869154091s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998678824 (496h47m3.869761869s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999124643 (496h47m3.870207688s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999669781 (496h47m3.870752826s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.000416904 (496h47m3.871499959s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002163089 (496h47m3.873246134s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002923166 (496h47m3.874006221s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.003684706 (496h47m3.874767771s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004377015 (496h47m3.87546007s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004494839 (496h47m3.875577884s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.005322004 (496h47m3.876405049s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.006274337 (496h47m3.877357372s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:05.709486098 (496h47m5.580569143s) [upload=436031]: Next returned nil segment Dec 26 08:16:05.709504011 (496h47m5.580587046s) [upload=436031]: Waiting for error group managing segments... Dec 26 08:16:05.997101825 (496h47m5.86818486s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.002826386 (496h47m5.873909431s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.003169179 (496h47m5.874252244s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.015164881 (496h47m5.886247916s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.017734023 (496h47m5.888817068s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.018932005 (496h47m5.89001504s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.020254483 (496h47m5.891337528s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.024683606 (496h47m5.895766651s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036334721 (496h47m5.907417756s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036433650 (496h47m5.907516695s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.251058020 (496h47m6.122141065s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.355066080 (496h47m6.226149115s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376711365 (496h47m6.2477944s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376731875 (496h47m6.24781491s) [seg_pos={0 0}, upload=436031]: Segment reached optimal threshold of 13 pieces. Dec 26 08:16:06.376924802 (496h47m6.248007837s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection Dec 26 08:16:06.376932617 (496h47m6.248015652s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.376963205 (496h47m6.24804625s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 08:16:06.376973816 (496h47m6.248056861s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377029190 (496h47m6.248112236s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection Dec 26 08:16:06.377035894 (496h47m6.248118929s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377050131 (496h47m6.248133166s) [seg_pos={0 0}, upload=436031]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:16:06.377121917 (496h47m6.248204952s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:16:06.377127738 (496h47m6.248210773s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:16:06.671006954 (496h47m6.542089989s) [upload=436031]: Done uploading segments... Dec 26 08:16:06.671028706 (496h47m6.542111741s) [upload=436031]: Upload finished. err: <nil> Duration: 6.072209555s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.471084627s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:17:05 UTC) orbiter warning: context deadline exceeded Duration: 3.471153538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X...X.....X.....XXX.X......................X..X.XX.X.X..X] duration: 30.002827777s trace: https://jaeger.dev.storj.io/trace/44b8d37bbaf1cbd0 FAIL (2025-12-26 08:16:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA) context deadline exceeded ======LOGS====== 82 (496h47m3.620943537s) [upload=436031]: Uploading segments... Dec 26 08:16:03.750170222 (496h47m3.621253337s) [upload=436031]: Got next segment. Inline: false Dec 26 08:16:03.750217301 (496h47m3.621300336s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:16:03.750225136 (496h47m3.621308181s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:16:03.996028027 (496h47m3.867111082s) [seg_pos={0 0}, upload=436031]: Begin upload segment... Dec 26 08:16:03.996201377 (496h47m3.867284422s) [seg_pos={0 0}, upload=436031]: Done begin upload segment. Dec 26 08:16:03.996260529 (496h47m3.867343574s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996356201 (496h47m3.867439246s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.996987703 (496h47m3.868070749s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.997416861 (496h47m3.868499906s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998071046 (496h47m3.869154091s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.998678824 (496h47m3.869761869s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999124643 (496h47m3.870207688s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:03.999669781 (496h47m3.870752826s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.000416904 (496h47m3.871499959s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002163089 (496h47m3.873246134s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.002923166 (496h47m3.874006221s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.003684706 (496h47m3.874767771s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004377015 (496h47m3.87546007s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.004494839 (496h47m3.875577884s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.005322004 (496h47m3.876405049s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:04.006274337 (496h47m3.877357372s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Uploading piece... Dec 26 08:16:05.709486098 (496h47m5.580569143s) [upload=436031]: Next returned nil segment Dec 26 08:16:05.709504011 (496h47m5.580587046s) [upload=436031]: Waiting for error group managing segments... Dec 26 08:16:05.997101825 (496h47m5.86818486s) [noise=true , address=145.79.227.41:4086 , piece_id=ZP4IHYILWOWMTZ7JQVDKTL46RQNAMPYKQJV2IGQY24XBXHWLSTXQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.002826386 (496h47m5.873909431s) [noise=true , address=145.79.227.50:4375 , piece_id=TMJRZHIBJYKYYBQIBV7FNSNRSCVKK6DRWK2IQWANHQ7TCXLWJUBA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.003169179 (496h47m5.874252244s) [noise=true , address=145.79.227.42:4123 , piece_id=MR6XRXMLCYFDLXYJBYK6WFGYS4UL7Y2JY2K7O6TWGI6Y5SDBYSLA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.015164881 (496h47m5.886247916s) [noise=true , address=145.79.227.6:4258 , piece_id=2NVYYMHFPAHCF4R3EL6WUQYKCV7ISQERD3AX33JXRZKMKRVRG7ZQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.017734023 (496h47m5.888817068s) [noise=true , address=145.79.227.52:4447 , piece_id=7GJUZMQ332Z5YAYZ3UZNNGNLJTDVN4U5R3CNEOPVYSGF6KU7VXGQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.018932005 (496h47m5.89001504s) [noise=true , address=145.79.227.11:4730 , piece_id=6DZGVD7ONGGQFEP2AWWZEPWSQK35LX7R3WIJHYWZHTLF6VO22PTA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.020254483 (496h47m5.891337528s) [noise=true , address=145.79.227.39:4014 , piece_id=OSEEUPEXSD6OQK26KN573V23MIVM52DXU66HISLZRB4PEAHULWWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.024683606 (496h47m5.895766651s) [noise=true , address=145.79.227.51:4410 , piece_id=UNM5VUNOUFVLRR4MEKR3DVHKBOPQ7AJWODTJFNWFDLW3HXZHPZKQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036334721 (496h47m5.907417756s) [noise=true , address=145.79.227.17:4970 , piece_id=3EK3VVPTJMI6VTKQX4UQXOQ7SYAU4EP25LMDQMSXMWB7YIBEN6EQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.036433650 (496h47m5.907516695s) [noise=true , address=145.79.227.29:4942 , piece_id=OSWQIFHEGH4GACNU3S3YRFJ7ELGABLDX2MKJJHCR6ELFBJPTDZOA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.251058020 (496h47m6.122141065s) [noise=true , address=45.91.116.14:4191 , piece_id=ISVXOICGV2H2KJFQPD72QJXYUWX3TPXINJ4MPWWNTTG24V4CGERA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.355066080 (496h47m6.226149115s) [noise=true , address=45.91.116.11:4120 , piece_id=QEP3Y7PVHUWB5EJ2OMG6VZLXVKSHIIA7IONGRVUL3LDVHIA52JWA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376711365 (496h47m6.2477944s) [noise=true , address=45.91.116.6:4973 , piece_id=SGRLJBEUVHLIJVRAUPSYMUGDSEO5C5DQVQNPQGREEGFOZZOK7ATQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: <nil> Dec 26 08:16:06.376731875 (496h47m6.24781491s) [seg_pos={0 0}, upload=436031]: Segment reached optimal threshold of 13 pieces. Dec 26 08:16:06.376924802 (496h47m6.248007837s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60986->77.90.31.13:4627: use of closed network connection Dec 26 08:16:06.376932617 (496h47m6.248015652s) [noise=true , address=77.90.31.13:4627 , piece_id=MSY7IO3EMFQN2X4TUZCUTYIRYFOJFQ7FSKXASK6ANWI3R4KNZXAQ, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.376963205 (496h47m6.24804625s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 08:16:06.376973816 (496h47m6.248056861s) [noise=true , address=77.90.31.10:4520 , piece_id=J4YIMGEOOZ4U6W4ZS47NNKKKU4FZAWEVGBTRXIZNSSFE3DJBGL2Q, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377029190 (496h47m6.248112236s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58174->77.90.31.15:4700: use of closed network connection Dec 26 08:16:06.377035894 (496h47m6.248118929s) [noise=true , address=77.90.31.15:4700 , piece_id=CJPG5GHDZAGZHF7QDB77N2DFIZWSJGKT544IYWMEAHVV3FUTL5LA, seg_pos={0 0}, upload=436031]: Long tail cancellation due to optimal threshold Dec 26 08:16:06.377050131 (496h47m6.248133166s) [seg_pos={0 0}, upload=436031]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:16:06.377121917 (496h47m6.248204952s) [upload=436031]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:16:06.377127738 (496h47m6.248210773s) [upload=436031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:16:06.671006954 (496h47m6.542089989s) [upload=436031]: Done uploading segments... Dec 26 08:16:06.671028706 (496h47m6.542111741s) [upload=436031]: Upload finished. err: <nil> Duration: 30.00288253s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.920008137s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:16:33 UTC) orbiter warning: context deadline exceeded Duration: 5.920032903s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.471471601s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:16:28 UTC) orbiter warning: context deadline exceeded Duration: 3.47149687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.915534257s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:16:16 UTC) orbiter warning: context deadline exceeded Duration: 5.915568543s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X...X.....X.....XXX.X......................X..X.XX.X.X..] duration: 5.694763623s trace: https://jaeger.dev.storj.io/trace/5418ee64ef539afc FAIL (2025-12-26 08:15:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ) context deadline exceeded ======LOGS====== Dec 26 08:13:56.387591062 (496h44m56.258674097s) [upload=435998]: Starting upload Dec 26 08:13:56.387645205 (496h44m56.25872824s) [upload=435998]: Done starting upload Dec 26 08:13:56.387743602 (496h44m56.258826637s) [upload=435998]: Uploading segments... Dec 26 08:13:56.387897605 (496h44m56.258980671s) [upload=435998]: Got next segment. Inline: false Dec 26 08:13:56.387939104 (496h44m56.259022159s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:13:56.387946799 (496h44m56.259029844s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:13:56.632308335 (496h44m56.50339137s) [seg_pos={0 0}, upload=435998]: Begin upload segment... Dec 26 08:13:56.632367236 (496h44m56.503450271s) [seg_pos={0 0}, upload=435998]: Done begin upload segment. Dec 26 08:13:56.632416010 (496h44m56.503499045s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.632480863 (496h44m56.503563898s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633103479 (496h44m56.504186524s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633422737 (496h44m56.504505782s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633910335 (496h44m56.50499337s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634257015 (496h44m56.50534006s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634560162 (496h44m56.505643197s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634808494 (496h44m56.50589154s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634898035 (496h44m56.50598107s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635211773 (496h44m56.506294818s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635243203 (496h44m56.506326238s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635634247 (496h44m56.506717292s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636007919 (496h44m56.507090954s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636378253 (496h44m56.507461299s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636760842 (496h44m56.507843887s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.639144031 (496h44m56.510227066s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:58.350452979 (496h44m58.221536024s) [upload=435998]: Next returned nil segment Dec 26 08:13:58.350510920 (496h44m58.221593975s) [upload=435998]: Waiting for error group managing segments... Dec 26 08:13:58.636874774 (496h44m58.50795782s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.645244031 (496h44m58.516327076s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.648582838 (496h44m58.519665883s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.650931201 (496h44m58.522014236s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.652815889 (496h44m58.523898924s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.654005154 (496h44m58.525088189s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.660084860 (496h44m58.531167895s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.667454473 (496h44m58.538537508s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.674669892 (496h44m58.545752927s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.676972116 (496h44m58.548055151s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.972357689 (496h44m58.843440725s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.976407091 (496h44m58.847490136s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995564359 (496h44m58.866647394s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995585260 (496h44m58.866668305s) [seg_pos={0 0}, upload=435998]: Segment reached optimal threshold of 13 pieces. Dec 26 08:13:58.995849242 (496h44m58.866932307s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995859150 (496h44m58.866942185s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.995890881 (496h44m58.866973946s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995900249 (496h44m58.866983284s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996085993 (496h44m58.867169048s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.996094178 (496h44m58.867177213s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996114267 (496h44m58.867197302s) [seg_pos={0 0}, upload=435998]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:13:58.996199899 (496h44m58.867282934s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:13:58.996207173 (496h44m58.867290218s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:13:59.287409842 (496h44m59.158492887s) [upload=435998]: Done uploading segments... Dec 26 08:13:59.287430241 (496h44m59.158513276s) [upload=435998]: Upload finished. err: <nil> Duration: 5.694786106s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.485047854s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:15:36 UTC) orbiter warning: context deadline exceeded Duration: 3.485076608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.904032426s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:15:13 UTC) orbiter warning: context deadline exceeded Duration: 5.904101166s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.432165668s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:15:11 UTC) orbiter warning: context deadline exceeded Duration: 3.432186998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X....X...X.....X.....XXX.X......................X..X.XX.X.X.] duration: 5.999915594s trace: https://jaeger.dev.storj.io/trace/5418ee64ef539afc FAIL (2025-12-26 08:15:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ) context deadline exceeded ======LOGS====== Dec 26 08:13:56.387591062 (496h44m56.258674097s) [upload=435998]: Starting upload Dec 26 08:13:56.387645205 (496h44m56.25872824s) [upload=435998]: Done starting upload Dec 26 08:13:56.387743602 (496h44m56.258826637s) [upload=435998]: Uploading segments... Dec 26 08:13:56.387897605 (496h44m56.258980671s) [upload=435998]: Got next segment. Inline: false Dec 26 08:13:56.387939104 (496h44m56.259022159s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:13:56.387946799 (496h44m56.259029844s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:13:56.632308335 (496h44m56.50339137s) [seg_pos={0 0}, upload=435998]: Begin upload segment... Dec 26 08:13:56.632367236 (496h44m56.503450271s) [seg_pos={0 0}, upload=435998]: Done begin upload segment. Dec 26 08:13:56.632416010 (496h44m56.503499045s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.632480863 (496h44m56.503563898s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633103479 (496h44m56.504186524s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633422737 (496h44m56.504505782s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633910335 (496h44m56.50499337s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634257015 (496h44m56.50534006s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634560162 (496h44m56.505643197s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634808494 (496h44m56.50589154s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634898035 (496h44m56.50598107s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635211773 (496h44m56.506294818s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635243203 (496h44m56.506326238s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635634247 (496h44m56.506717292s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636007919 (496h44m56.507090954s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636378253 (496h44m56.507461299s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636760842 (496h44m56.507843887s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.639144031 (496h44m56.510227066s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:58.350452979 (496h44m58.221536024s) [upload=435998]: Next returned nil segment Dec 26 08:13:58.350510920 (496h44m58.221593975s) [upload=435998]: Waiting for error group managing segments... Dec 26 08:13:58.636874774 (496h44m58.50795782s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.645244031 (496h44m58.516327076s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.648582838 (496h44m58.519665883s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.650931201 (496h44m58.522014236s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.652815889 (496h44m58.523898924s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.654005154 (496h44m58.525088189s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.660084860 (496h44m58.531167895s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.667454473 (496h44m58.538537508s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.674669892 (496h44m58.545752927s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.676972116 (496h44m58.548055151s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.972357689 (496h44m58.843440725s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.976407091 (496h44m58.847490136s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995564359 (496h44m58.866647394s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995585260 (496h44m58.866668305s) [seg_pos={0 0}, upload=435998]: Segment reached optimal threshold of 13 pieces. Dec 26 08:13:58.995849242 (496h44m58.866932307s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995859150 (496h44m58.866942185s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.995890881 (496h44m58.866973946s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995900249 (496h44m58.866983284s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996085993 (496h44m58.867169048s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.996094178 (496h44m58.867177213s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996114267 (496h44m58.867197302s) [seg_pos={0 0}, upload=435998]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:13:58.996199899 (496h44m58.867282934s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:13:58.996207173 (496h44m58.867290218s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:13:59.287409842 (496h44m59.158492887s) [upload=435998]: Done uploading segments... Dec 26 08:13:59.287430241 (496h44m59.158513276s) [upload=435998]: Upload finished. err: <nil> Duration: 5.999943877s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.154531408s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:14:31 UTC) orbiter warning: context deadline exceeded Duration: 6.154576193s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX....X...X.....X.....XXX.X......................X..X.XX.X.X] duration: 30.003453606s trace: https://jaeger.dev.storj.io/trace/5418ee64ef539afc FAIL (2025-12-26 08:14:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ) context deadline exceeded ======LOGS====== Dec 26 08:13:56.387591062 (496h44m56.258674097s) [upload=435998]: Starting upload Dec 26 08:13:56.387645205 (496h44m56.25872824s) [upload=435998]: Done starting upload Dec 26 08:13:56.387743602 (496h44m56.258826637s) [upload=435998]: Uploading segments... Dec 26 08:13:56.387897605 (496h44m56.258980671s) [upload=435998]: Got next segment. Inline: false Dec 26 08:13:56.387939104 (496h44m56.259022159s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:13:56.387946799 (496h44m56.259029844s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:13:56.632308335 (496h44m56.50339137s) [seg_pos={0 0}, upload=435998]: Begin upload segment... Dec 26 08:13:56.632367236 (496h44m56.503450271s) [seg_pos={0 0}, upload=435998]: Done begin upload segment. Dec 26 08:13:56.632416010 (496h44m56.503499045s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.632480863 (496h44m56.503563898s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633103479 (496h44m56.504186524s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633422737 (496h44m56.504505782s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.633910335 (496h44m56.50499337s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634257015 (496h44m56.50534006s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634560162 (496h44m56.505643197s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634808494 (496h44m56.50589154s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.634898035 (496h44m56.50598107s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635211773 (496h44m56.506294818s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635243203 (496h44m56.506326238s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.635634247 (496h44m56.506717292s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636007919 (496h44m56.507090954s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636378253 (496h44m56.507461299s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.636760842 (496h44m56.507843887s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:56.639144031 (496h44m56.510227066s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Uploading piece... Dec 26 08:13:58.350452979 (496h44m58.221536024s) [upload=435998]: Next returned nil segment Dec 26 08:13:58.350510920 (496h44m58.221593975s) [upload=435998]: Waiting for error group managing segments... Dec 26 08:13:58.636874774 (496h44m58.50795782s) [noise=true , address=145.79.227.6:4257 , piece_id=Z2F6SFAXWTCAIFP25CAB3CSGNG7H6KSKLW6XWK4356RH2RXEWNEQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.645244031 (496h44m58.516327076s) [noise=true , address=145.79.227.41:4087 , piece_id=I5MODW6CJ25SWBL54DOGV33M4LXWRNZSLCSWTMVIP63KQL3BMB2A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.648582838 (496h44m58.519665883s) [noise=true , address=145.79.227.18:4006 , piece_id=SSEBI4E2XEH4XNH3UUI5A2T6WH64PQWV2QHJMUB3RX2ENALNXERA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.650931201 (496h44m58.522014236s) [noise=true , address=145.79.227.52:4446 , piece_id=EO6WEDJGW762BFYEMXHUJB3QHHEKF7K5APN24I5JX4KQPJU42ABQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.652815889 (496h44m58.523898924s) [noise=true , address=145.79.227.36:4517 , piece_id=AMJBQ3XSC2RWVHZGHIUYEDUFSCOJVBHUSTN7OX5TDMWFXJ3K45AQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.654005154 (496h44m58.525088189s) [noise=true , address=145.79.227.5:4223 , piece_id=PY2PGPR2URJ42ZSICS54OESSUNDWL5HQYW7SDLTAXA4SYSJTUSDA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.660084860 (496h44m58.531167895s) [noise=true , address=145.79.227.32:4835 , piece_id=AWRO2OSU54WUOOHXXSYAA6Z7EIJRQJCEYILUU6JNVCJRWLNIBHCQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.667454473 (496h44m58.538537508s) [noise=true , address=145.79.227.46:4603 , piece_id=HDVP4MIQ3LXDIGZCQJPCHWNY6PPMGZQBQNYWXHYRZA7XBCCJK4WQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.674669892 (496h44m58.545752927s) [noise=true , address=145.79.227.4:4187 , piece_id=HASYJJ5JRWIKUTRRL7LFGZ5X6DUGSKRAW4YBX4Q4YJOT5UBZW2OA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.676972116 (496h44m58.548055151s) [noise=true , address=145.79.227.47:4265 , piece_id=KDQJJ4NYVQJWHGTWE35Y7HZBNYT76BZE5XEXT3HYHCW6RP23BVEA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.972357689 (496h44m58.843440725s) [noise=true , address=45.91.116.5:4300 , piece_id=L4RD7FQ33TFIEUORLDP5DUCQHCSWNEIUQ6ZSYLQM5TEYZI2YNLTA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.976407091 (496h44m58.847490136s) [noise=true , address=45.91.116.9:4262 , piece_id=E4PJNIU3IT7O7FB7RGF2GNKPFGEST5OR657AYHQ2FQNAMNB67MIA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995564359 (496h44m58.866647394s) [noise=true , address=45.91.116.11:4120 , piece_id=4PNQADIHDPA4ZF65X5WWBD3BIY2N53AFWDPLISOETXTCAJODCOVQ, seg_pos={0 0}, upload=435998]: Done uploading piece. err: <nil> Dec 26 08:13:58.995585260 (496h44m58.866668305s) [seg_pos={0 0}, upload=435998]: Segment reached optimal threshold of 13 pieces. Dec 26 08:13:58.995849242 (496h44m58.866932307s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995859150 (496h44m58.866942185s) [noise=true , address=45.91.116.12:4225 , piece_id=MDEI5W3I3V36VIW463XJREQSNFZ5QRGMM2TKW37GD4OUY5UNBYNA, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.995890881 (496h44m58.866973946s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.995900249 (496h44m58.866983284s) [noise=true , address=77.90.31.18:4808 , piece_id=EYNK63ZS2BFGVNHEAKMMWQNEVSDDLMJJE7LTDT7SRBQOJTTUYL5A, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996085993 (496h44m58.867169048s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 08:13:58.996094178 (496h44m58.867177213s) [noise=true , address=145.79.227.44:4195 , piece_id=MKUCVJOPI2TK2Z4K2NH4TDVQ55HE3NWAT5PF4JEPS7IJUAWBTT4Q, seg_pos={0 0}, upload=435998]: Long tail cancellation due to optimal threshold Dec 26 08:13:58.996114267 (496h44m58.867197302s) [seg_pos={0 0}, upload=435998]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:13:58.996199899 (496h44m58.867282934s) [upload=435998]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:13:58.996207173 (496h44m58.867290218s) [upload=435998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:13:59.287409842 (496h44m59.158492887s) [upload=435998]: Done uploading segments... Dec 26 08:13:59.287430241 (496h44m59.158513276s) [upload=435998]: Upload finished. err: <nil> Duration: 30.003476721s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.974683161s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:14:11 UTC) orbiter warning: context deadline exceeded Duration: 3.974759267s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 9.840040133s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:14:00 UTC) orbiter warning: context deadline exceeded Duration: 9.840122488s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.290595178s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:13:35 UTC) orbiter warning: context deadline exceeded Duration: 6.290619934s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.369206274s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:13:25 UTC) orbiter warning: context deadline exceeded Duration: 3.369230479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XX....X...X.....X.....XXX.X......................X..X.XX.X.] duration: 6.088908241s trace: https://jaeger.dev.storj.io/trace/17d9d640ae0e3b3c FAIL (2025-12-26 08:12:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:11:43.193456026 (496h42m43.064539071s) [upload=435966]: Starting upload Dec 26 08:11:43.193520309 (496h42m43.064603344s) [upload=435966]: Done starting upload Dec 26 08:11:43.193652050 (496h42m43.064735105s) [upload=435966]: Uploading segments... Dec 26 08:11:43.193724478 (496h42m43.064807523s) [upload=435966]: Got next segment. Inline: false Dec 26 08:11:43.193821351 (496h42m43.064904386s) [upload=435966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:11:43.193849375 (496h42m43.06493243s) [upload=435966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:11:43.441602889 (496h42m43.312685924s) [seg_pos={0 0}, upload=435966]: Begin upload segment... Dec 26 08:11:43.441667572 (496h42m43.312750617s) [seg_pos={0 0}, upload=435966]: Done begin upload segment. Dec 26 08:11:43.441694123 (496h42m43.312777168s) [noise=true , address=145.79.227.36:4517 , piece_id=XF5NYT3EQGEPUADYXICF6TXGKVTOVK5C2JURNBYLC3N5EJHLEQIQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.441718871 (496h42m43.312801906s) [noise=true , address=45.91.116.14:4189 , piece_id=G5J5G7KEPXPOFBTLYZ3SEWFDI25J56SWMNSUD5HGS5B7VOK7ZUSQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.442435905 (496h42m43.31351894s) [noise=true , address=145.79.227.19:4078 , piece_id=D5D4NF4U5OU3VJDFF4UNRDUYN53NAWFAOXE4IYYXT3VNRX32BWXQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.442444543 (496h42m43.313527588s) [noise=true , address=45.91.116.8:4048 , piece_id=GIE7RPBX3COJSJQJPV25K4AFJIBASMHXMR73RDYZUAU6XVNPIEHQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.442955134 (496h42m43.314038179s) [noise=true , address=145.79.227.41:4086 , piece_id=BH3IWHNWFDETI7E6BOIQRYXFYTXPOQSGYLFSQ77UO67NWQQIARRA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443024216 (496h42m43.314107261s) [noise=true , address=145.79.227.6:4257 , piece_id=PKSECFXIUAVZ5UWMMUHOWAWCFW7XUEGQJB6ZARVY3Q5LP4MNBGZQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443372629 (496h42m43.314455674s) [noise=true , address=145.79.227.3:4150 , piece_id=LKKAAOJXZ6EJLTRDZKE5EFTAD4IFZORFILMMAVQORMKU7B2JO5YA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443532684 (496h42m43.314615729s) [noise=true , address=145.79.227.51:4410 , piece_id=VB7VSJNUWT6LWOHL5NM37VDAOA4L2B7OTX3IF4LSNJWV5WRSQEGA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.443967452 (496h42m43.315050497s) [noise=true , address=77.90.31.11:4556 , piece_id=TOFFKBK4XGH5Q2JKI37TQFWJMQY6QM46CGIFBLW5RS3P2JJAUQRA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.444374397 (496h42m43.315457442s) [noise=true , address=145.79.227.52:4446 , piece_id=EG3ICHGG3RZBPCUELATHMFEXWNHHXTICCSCSFUCI67O5QN33PTEQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.444704516 (496h42m43.315787551s) [noise=true , address=45.91.116.4:4937 , piece_id=NDWW4K4DDSPVOCPHQK7I7JCWZFT7344H2HNSIR4FEI52E6QZMOJA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.444924875 (496h42m43.31600792s) [noise=true , address=145.79.227.9:4367 , piece_id=3K2C6ZGHLYOHWLUQYFLLDHJXJB5H6BX4XOS67MICU3XM7BUGWLNA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.445350766 (496h42m43.316433811s) [noise=true , address=145.79.227.13:4803 , piece_id=JEW25O7EGLEYAADLG33ASB2O2JKVM5DCGZBGFA4A3NKWT33AJQKQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.445705341 (496h42m43.316788376s) [noise=true , address=145.79.227.16:4910 , piece_id=25WEOZ5MZLUI7OSJPAKWCQOMYUCO6RTP7P65W5U7FSBHTSDTQGUA, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.446028307 (496h42m43.317111342s) [noise=true , address=45.91.116.6:4976 , piece_id=YMZNPO55XMKGDL6RNFHFOSTVGRYG4BZLF5Z4HNHXKKWLR2X4CHXQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:43.446432155 (496h42m43.3175152s) [noise=true , address=145.79.227.38:4979 , piece_id=C2EG7623HRJLTJW2QR4RVIOJ4JKPLRZM2NJHWDC7FNC32SX7RPKQ, seg_pos={0 0}, upload=435966]: Uploading piece... Dec 26 08:11:45.153862759 (496h42m45.024945794s) [upload=435966]: Next returned nil segment Dec 26 08:11:45.153886234 (496h42m45.024969269s) [upload=435966]: Waiting for error group managing segments... Dec 26 08:11:45.448693466 (496h42m45.319776501s) [noise=true , address=145.79.227.51:4410 , piece_id=VB7VSJNUWT6LWOHL5NM37VDAOA4L2B7OTX3IF4LSNJWV5WRSQEGA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.452290074 (496h42m45.323373109s) [noise=true , address=145.79.227.36:4517 , piece_id=XF5NYT3EQGEPUADYXICF6TXGKVTOVK5C2JURNBYLC3N5EJHLEQIQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.453739825 (496h42m45.32482287s) [noise=true , address=145.79.227.38:4979 , piece_id=C2EG7623HRJLTJW2QR4RVIOJ4JKPLRZM2NJHWDC7FNC32SX7RPKQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.454108528 (496h42m45.325191563s) [noise=true , address=145.79.227.9:4367 , piece_id=3K2C6ZGHLYOHWLUQYFLLDHJXJB5H6BX4XOS67MICU3XM7BUGWLNA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.456227342 (496h42m45.327310377s) [noise=true , address=145.79.227.3:4150 , piece_id=LKKAAOJXZ6EJLTRDZKE5EFTAD4IFZORFILMMAVQORMKU7B2JO5YA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.457784807 (496h42m45.328867842s) [noise=true , address=145.79.227.13:4803 , piece_id=JEW25O7EGLEYAADLG33ASB2O2JKVM5DCGZBGFA4A3NKWT33AJQKQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.471772232 (496h42m45.342855277s) [noise=true , address=145.79.227.6:4257 , piece_id=PKSECFXIUAVZ5UWMMUHOWAWCFW7XUEGQJB6ZARVY3Q5LP4MNBGZQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.472312932 (496h42m45.343395977s) [noise=true , address=145.79.227.16:4910 , piece_id=25WEOZ5MZLUI7OSJPAKWCQOMYUCO6RTP7P65W5U7FSBHTSDTQGUA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.477544674 (496h42m45.348627709s) [noise=true , address=145.79.227.41:4086 , piece_id=BH3IWHNWFDETI7E6BOIQRYXFYTXPOQSGYLFSQ77UO67NWQQIARRA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.491847869 (496h42m45.362930904s) [noise=true , address=145.79.227.52:4446 , piece_id=EG3ICHGG3RZBPCUELATHMFEXWNHHXTICCSCSFUCI67O5QN33PTEQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.501135975 (496h42m45.37221902s) [noise=true , address=145.79.227.19:4078 , piece_id=D5D4NF4U5OU3VJDFF4UNRDUYN53NAWFAOXE4IYYXT3VNRX32BWXQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.608631485 (496h42m45.47971453s) [noise=true , address=77.90.31.11:4556 , piece_id=TOFFKBK4XGH5Q2JKI37TQFWJMQY6QM46CGIFBLW5RS3P2JJAUQRA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.728710294 (496h42m45.599793339s) [noise=true , address=45.91.116.6:4976 , piece_id=YMZNPO55XMKGDL6RNFHFOSTVGRYG4BZLF5Z4HNHXKKWLR2X4CHXQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: <nil> Dec 26 08:11:45.728735372 (496h42m45.599818417s) [seg_pos={0 0}, upload=435966]: Segment reached optimal threshold of 13 pieces. Dec 26 08:11:45.729023932 (496h42m45.600106988s) [noise=true , address=45.91.116.4:4937 , piece_id=NDWW4K4DDSPVOCPHQK7I7JCWZFT7344H2HNSIR4FEI52E6QZMOJA, seg_pos={0 0}, upload=435966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:36580->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:11:45.729034102 (496h42m45.600117147s) [noise=true , address=45.91.116.4:4937 , piece_id=NDWW4K4DDSPVOCPHQK7I7JCWZFT7344H2HNSIR4FEI52E6QZMOJA, seg_pos={0 0}, upload=435966]: Long tail cancellation due to optimal threshold Dec 26 08:11:45.729235996 (496h42m45.600319061s) [noise=true , address=45.91.116.14:4189 , piece_id=G5J5G7KEPXPOFBTLYZ3SEWFDI25J56SWMNSUD5HGS5B7VOK7ZUSQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 08:11:45.729246326 (496h42m45.600329371s) [noise=true , address=45.91.116.14:4189 , piece_id=G5J5G7KEPXPOFBTLYZ3SEWFDI25J56SWMNSUD5HGS5B7VOK7ZUSQ, seg_pos={0 0}, upload=435966]: Long tail cancellation due to optimal threshold Dec 26 08:11:45.729268638 (496h42m45.600351673s) [noise=true , address=45.91.116.8:4048 , piece_id=GIE7RPBX3COJSJQJPV25K4AFJIBASMHXMR73RDYZUAU6XVNPIEHQ, seg_pos={0 0}, upload=435966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 08:11:45.729279359 (496h42m45.600362394s) [noise=true , address=45.91.116.8:4048 , piece_id=GIE7RPBX3COJSJQJPV25K4AFJIBASMHXMR73RDYZUAU6XVNPIEHQ, seg_pos={0 0}, upload=435966]: Long tail cancellation due to optimal threshold Dec 26 08:11:45.729311299 (496h42m45.600394344s) [seg_pos={0 0}, upload=435966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:11:45.729417842 (496h42m45.600500887s) [upload=435966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:11:45.729425286 (496h42m45.600508331s) [upload=435966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:11:46.015994742 (496h42m45.887077777s) [upload=435966]: Done uploading segments... Dec 26 08:11:46.016026893 (496h42m45.887109928s) [upload=435966]: Upload finished. err: <nil> Duration: 6.088936714s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)