Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.827663837s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:34:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.82768094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.696851246s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:34:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.696873198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.009328345s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:34:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.009353221s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.781823589s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:34:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.781841543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................X.......X........X............X.X.........] duration: 5.656923928s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:34:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.656954896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X.X...X.X.X....XX..........................................] duration: 3.536286324s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:34:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.536310159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.723068679s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:33:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.723087917s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................X.......X........X............X.X........] duration: 5.866227499s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:33:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.866253258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.995843636s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:33:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.995872001s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.790514949s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:33:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.790544224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.998338225s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:33:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.998367211s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X.X...X.X.X....XX.........................................] duration: 3.389194115s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:33:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.389215154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.702407288s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:33:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.702434549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.73089918s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:32:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.730921101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.012255908s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:32:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.012285473s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X.X...X.X.X....XX........................................] duration: 3.588403863s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:32:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.5884286s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................X.......X........X............X.X.......] duration: 5.672880332s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:32:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.672909888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.834099519s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:32:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.834113307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.731695989s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:31:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.731724281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.713357958s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:31:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.713382295s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................X.......X........X............X.X......] duration: 5.888234613s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:31:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.888264468s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X.X...X.X.X....XX.......................................] duration: 3.383281175s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:31:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.383309689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.418830668s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:31:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.418861789s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X.X...X.X.X....XX......................................] duration: 3.531363399s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:31:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531473209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.736639722s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:30:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.736688245s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.846525248s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:30:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.84655231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.047944895s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:30:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.047966345s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.724604716s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:30:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.724627818s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.810137676s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:30:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.810168674s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................X.......X........X............X.X.....] duration: 5.895713915s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:30:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.895747429s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X.X...X.X.X....XX.....................................] duration: 3.414272708s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:30:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.414314818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.826674623s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:29:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.826696074s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.019712099s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:29:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.019871553s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.578845946s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:29:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.578871354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.979768798s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:29:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.979797362s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................X.......X........X............X.X....] duration: 5.732097323s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:29:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.73212786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.839353445s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:29:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.83938288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.......X.X...X.X.X....XX....................................] duration: 3.528489516s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:28:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528516947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.991911981s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:28:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.991940785s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.639195642s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:28:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.639218255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..............................................X............] duration: 5.896740641s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:28:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.896769355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..............................................X...........] duration: 5.839169097s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:28:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.839200455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........................X.......X........X............X.X...] duration: 5.678199019s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:28:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.678303748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.710686281s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:28:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.71071211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.002098219s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:28:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.002126653s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [........X.X...X.X.X....XX...................................] duration: 3.434423428s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:27:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.434655109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.468035122s trace: https://jaeger.dev.storj.io/trace/44eba3f11f79732e FAIL (2025-12-09 01:27:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.468059899s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..............................................X..........] duration: 5.860121821s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:27:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.860153182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.986988455s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:27:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.987015206s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........................X.......X........X............X.X..] duration: 5.839791589s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:27:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.839814483s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)