Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........X.X...X.X.X....XX..................................] duration: 3.585300112s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:27:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.585359656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.668601816s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:27:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.668629678s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.034788273s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:26:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.034820366s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..............................................X.........] duration: 5.694282465s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:26:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.694311931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.160752032s trace: https://jaeger.dev.storj.io/trace/44eba3f11f79732e FAIL (2025-12-09 01:26:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.16077719s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.991600387s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:26:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.991628521s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................X.......X........X............X.X.] duration: 5.660270278s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:26:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.660284155s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........................X.......X........X............X.X] duration: 1m0.001758169s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:26:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001783456s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X.........X.X...X.X.X....XX.................................] duration: 3.572892737s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:26:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.572917815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.727888016s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:26:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.727913806s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.156186219s trace: https://jaeger.dev.storj.io/trace/44eba3f11f79732e FAIL (2025-12-09 01:25:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.156220244s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..............................................X........] duration: 5.85520558s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:25:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.855235808s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X.........X.X...X.X.X....XX................................] duration: 3.479053301s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:25:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479079861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 5.301420448s trace: https://jaeger.dev.storj.io/trace/44eba3f11f79732e FAIL (2025-12-09 01:25:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.301445938s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.969545149s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:25:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.969574006s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.713493324s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:25:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.713524494s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................X.......X........X............X.] duration: 5.873256644s trace: https://jaeger.dev.storj.io/trace/273264eb5e94583f FAIL (2025-12-09 01:25:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.873275931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.672593424s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:24:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.672622711s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................X.......X........X............X] duration: 1m0.001620391s trace: https://jaeger.dev.storj.io/trace/273264eb5e94583f FAIL (2025-12-09 01:24:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.0016458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..............................................X.......] duration: 5.785693998s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:24:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.785720899s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X.........X.X...X.X.X....XX...............................] duration: 3.60548593s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:24:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.605513483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration: 4.861274256s trace: https://jaeger.dev.storj.io/trace/44eba3f11f79732e FAIL (2025-12-09 01:24:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.861307459s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.687675395s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:24:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.687698971s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.001057215s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:24:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.00108024s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..X......................................................] duration: 4.928475578s trace: https://jaeger.dev.storj.io/trace/44eba3f11f79732e FAIL (2025-12-09 01:24:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.928506117s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X.........X.X...X.X.X....XX..............................] duration: 3.48486051s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:24:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484890256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..............................................X......] duration: 5.896857491s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:23:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.896890343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.770225595s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:23:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.770253118s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....X.........X.X...X.X.X....XX.............................] duration: 3.508052332s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:23:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508079593s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 475.499342ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6b1ae90bcdd087c7 FAIL (2025-12-09 01:23:50 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.527675ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.682306309s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:23:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.682332801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.024411765s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:23:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.024440661s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......X........X............] duration: 5.943612644s trace: https://jaeger.dev.storj.io/trace/1414775820c29638 FAIL (2025-12-09 01:23:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.943650836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..X.....................................................] duration: 4.943068986s trace: https://jaeger.dev.storj.io/trace/44eba3f11f79732e FAIL (2025-12-09 01:23:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.943099545s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.026934117s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:23:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.027182862s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....X.........X.X...X.X.X....XX............................] duration: 3.563040957s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:23:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563065875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..X....................................................] duration: 5.265393268s trace: https://jaeger.dev.storj.io/trace/44eba3f11f79732e FAIL (2025-12-09 01:23:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.265418175s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 473.436964ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6b1ae90bcdd087c7 FAIL (2025-12-09 01:22:53 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 473.464707ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......X.........X.X...X.X.X....XX...........................] duration: 3.555545093s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:22:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555568178s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..............................................X.....] duration: 5.809835507s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:22:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.809869173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.002261647s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:22:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.002291646s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 475.070606ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6b1ae90bcdd087c7 FAIL (2025-12-09 01:22:28 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.101315ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.699290462s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:22:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.699322783s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.......X........X...........] duration: 5.712281239s trace: https://jaeger.dev.storj.io/trace/1414775820c29638 FAIL (2025-12-09 01:22:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.712316235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.709289149s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:21:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.709307764s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..X...................................................] duration: 5.367373256s trace: https://jaeger.dev.storj.io/trace/44eba3f11f79732e FAIL (2025-12-09 01:21:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.367405126s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 472.316126ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6b1ae90bcdd087c7 FAIL (2025-12-09 01:21:47 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 472.336485ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..............................................X....] duration: 5.724865619s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:21:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.724905726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................X.......X........X..........] duration: 6.009252219s trace: https://jaeger.dev.storj.io/trace/1414775820c29638 FAIL (2025-12-09 01:21:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.009275321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......X.........X.X...X.X.X....XX..........................] duration: 3.470552232s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:21:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470573962s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)