Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.001451304s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:49:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.001477754s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.728627247s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:49:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.728656313s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.852775778s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:49:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.85280285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.550592641s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:49:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.550607058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.606537541s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:49:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.606591965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X........X............X.X.............................] duration: 5.599846249s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:48:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.599937472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.993521964s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:48:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.993553405s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.670392441s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:48:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.670443478s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.542510815s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:48:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542572632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X........X............X.X............................] duration: 5.691136478s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:48:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.691300441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.627726877s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:48:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.627755952s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.985529894s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:48:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.985555393s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.835712917s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:47:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.83574055s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.535436112s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:47:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53547249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.015168032s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:47:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.015198088s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X.......X........X............X.X...........................] duration: 5.781125852s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:47:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.781152113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.92368519s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:47:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.923709115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.715974294s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:46:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.715999873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.966062307s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:46:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.966095438s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.46950964s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:46:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469539696s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.040984449s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:46:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.041014367s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X.......X........X............X.X..........................] duration: 5.758137111s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:46:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.758161236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.702745105s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:46:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.702771204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.035830888s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:46:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.035861906s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X...................................................X.......] duration: 3.541076725s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:45:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.541104497s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..X.......X........X............X.X.........................] duration: 5.80811707s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:45:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.808145605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XX...................................................X......] duration: 3.697899008s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:45:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.697926711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.835611276s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:45:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.835640281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.898721226s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:45:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.898746705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.028905369s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:45:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.028923804s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X.......X........X............X.X........................] duration: 5.832164704s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:44:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.832192456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.076692163s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:44:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.076724946s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.XX...................................................X.....] duration: 3.468082801s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:44:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468111346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.664081031s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:44:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.664120006s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.912714099s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:44:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.912759555s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....X.......X........X............X.X.......................] duration: 5.936436743s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:44:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.936465106s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..XX...................................................X....] duration: 3.468881532s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:44:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468907041s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.818282539s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:44:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.81831437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.673790885s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:43:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.673823305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.979807038s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:43:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.979835614s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X.......X........X............X.X......................] duration: 5.773169517s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:43:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.773187852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.993201593s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:43:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.993226671s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...XX...................................................X...] duration: 3.565429295s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:43:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.56545321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.789995143s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:43:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.790021443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.697281632s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:43:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.697312571s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X.......X........X............X.X.....................] duration: 6.216461377s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:43:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.216496494s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....XX...................................................X..] duration: 3.600992734s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:42:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.601018032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.984152354s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:42:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.98418154s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.870904498s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:42:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.870931911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......X.......X........X............X.X....................] duration: 5.74765513s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:42:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.747687643s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)