Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.7200619s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:42:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.720092297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.979055208s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:41:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.979084864s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X....XX...................................................X.] duration: 3.606230447s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:41:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.606266676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.632645302s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:41:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.632669087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.857535401s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:41:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.85758739s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........X.......X........X............X.X...................] duration: 5.813685252s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:41:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.813708727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.985094946s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:41:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.985122027s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X....XX...................................................X] duration: 5.001287001s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:40:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001314835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.X....XX...................................................] duration: 3.392081441s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:40:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.392107241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.705862773s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:40:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.70589247s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.855919714s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:40:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.855952136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.783631947s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:40:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.783665971s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X.X....XX..................................................] duration: 3.504289446s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:40:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.504339389s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........X.......X........X............X.X..................] duration: 6.155363583s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:40:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.155392238s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.047886193s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:39:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.0479169s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.70305707s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:39:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.703089561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.949529195s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:39:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.949554023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.X.X....XX.................................................] duration: 3.520895522s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:39:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520915509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.817949333s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:39:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.817981543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.00666936s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:39:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.006733772s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.697838467s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:38:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.697867481s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.871194956s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:38:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.871220255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X.X.X....XX................................................] duration: 3.374964408s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:38:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.374988815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........X.......X........X............X.X.................] duration: 5.657683394s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:38:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.657714473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.706367557s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:38:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.706387714s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........X.......X........X............X.X................] duration: 5.64786127s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:38:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.6478816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.007452522s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:38:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.0074778s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............X.......X........X............X.X...............] duration: 5.792646796s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:38:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.792679358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.847169077s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:38:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.847197221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.740692809s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:38:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.740715211s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X.X.X....XX...............................................] duration: 3.555313192s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:38:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555339263s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.969929112s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:37:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.969955242s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............X.......X........X............X.X..............] duration: 5.763278899s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:37:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.763303555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.700609554s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:37:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.700637698s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............X.......X........X............X.X.............] duration: 5.78923934s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:37:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.789267473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.977481954s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:36:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.977508004s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.828701172s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:36:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.82872613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X.X.X....XX..............................................] duration: 3.49368868s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:36:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.493712194s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............X.......X........X............X.X............] duration: 5.759144424s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:36:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.759240797s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.531020264s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:36:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.531040572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.722125705s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:36:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.722192662s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.976341042s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:35:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.97637192s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X...X.X.X....XX.............................................] duration: 3.392768801s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:35:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39279558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................X.......X........X............X.X...........] duration: 5.818055269s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:35:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.818181431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.706459813s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:35:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.706493026s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X...X.X.X....XX............................................] duration: 3.482306967s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:35:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.482332216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................X.......X........X............X.X..........] duration: 5.837666483s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:35:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.837697412s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.629891825s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:35:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.62991958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.800809601s trace: https://jaeger.dev.storj.io/trace/61e1f15b0de53a06 FAIL (2025-12-09 01:35:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 01:05:04.077327412 (81h36m3.948410457s) [upload=71299]: Starting upload Dec 9 01:05:04.077479422 (81h36m3.948562467s) [upload=71299]: Done starting upload Dec 9 01:05:04.077564383 (81h36m3.948647428s) [upload=71299]: Uploading segments... Dec 9 01:05:04.077964444 (81h36m3.94904751s) [upload=71299]: Got next segment. Inline: false Dec 9 01:05:04.078021453 (81h36m3.949104498s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 01:05:04.078029158 (81h36m3.949112203s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 01:05:04.325093769 (81h36m4.196176814s) [seg_pos={0 0}, upload=71299]: Begin upload segment... Dec 9 01:05:04.325151339 (81h36m4.196234384s) [seg_pos={0 0}, upload=71299]: Done begin upload segment. Dec 9 01:05:04.325193429 (81h36m4.196276474s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325308208 (81h36m4.196391253s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.325950250 (81h36m4.197033296s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326217961 (81h36m4.197301006s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.326682044 (81h36m4.197765089s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327265615 (81h36m4.19834867s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327577640 (81h36m4.198660695s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.327710382 (81h36m4.198793428s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328164918 (81h36m4.199247963s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.328390287 (81h36m4.199473332s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.329537922 (81h36m4.200620967s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.330251962 (81h36m4.201335017s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.331349081 (81h36m4.202432126s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.332272459 (81h36m4.203355504s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.333587923 (81h36m4.204670969s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:04.334281264 (81h36m4.205364319s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Uploading piece... Dec 9 01:05:06.034632558 (81h36m5.905715603s) [upload=71299]: Next returned nil segment Dec 9 01:05:06.034650011 (81h36m5.905733076s) [upload=71299]: Waiting for error group managing segments... Dec 9 01:05:06.319986357 (81h36m6.191069402s) [noise=true , address=145.79.227.22:4511 , piece_id=H7ZL44YXQPQ3MDIYFV2J5ID6SQAKGEDZ4QYIVJNVJVUJZXPEA5NQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.320738881 (81h36m6.191821926s) [noise=true , address=145.79.227.24:4583 , piece_id=XSMQMWNH4ZB2CKZVNFONMZI37UD7U7VYBT3NCXCOQJYVQ7KG73PQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.338113855 (81h36m6.20919689s) [noise=true , address=145.79.227.40:4050 , piece_id=TRRVNKZDEZOELFC2J5ILRGGVQ7FMFGNF7KY4K73PDHHEATNGNMAA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.341292378 (81h36m6.212375443s) [noise=true , address=145.79.227.25:4617 , piece_id=VAWH3IVQTJGP4RR635W7MUQQ6PE4JU65PI4F2RTNVEXHAQRQ7CZQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.344156311 (81h36m6.215239356s) [noise=true , address=145.79.227.31:4799 , piece_id=E6HPGN2U4POXUGUUZ7YJN5WDB7Q3YO34BXDQYVJD32PZSJ5APYKQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.351339228 (81h36m6.222422273s) [noise=true , address=145.79.227.22:4509 , piece_id=JRN2BT5YNORGFNP2JZJLIXCEYJOPNTIVLSHLGX4OCG5PCUQOFO3A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.368700567 (81h36m6.239783612s) [noise=true , address=145.79.227.20:4438 , piece_id=PCSNROLMD3UBUHJCGIKATYJJ4LC7Y433MHPFJOCQ2VEZRHI5OC4Q, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.377863905 (81h36m6.248947842s) [noise=true , address=145.79.227.29:4941 , piece_id=MELXRUKLN2YFQYIP6BSP5A2K35XRZGABISUEAIGGXFWJNRNTLGDA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379184129 (81h36m6.250267175s) [noise=true , address=145.79.227.16:4910 , piece_id=3MZ42IEHE6MEUZB2MFQCPT4TE5PJBLXKWHXISRKUSWXDA45P2NFQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.379479442 (81h36m6.250562487s) [noise=true , address=145.79.227.43:4159 , piece_id=DFUORMG2PBZFQXWO4WVYJENGJ7HTGHRAPGIH67LGEILW7BILL3HA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.381106200 (81h36m6.252189245s) [noise=true , address=145.79.227.40:4051 , piece_id=XHFYGRPQY7WNYXPX6CY5AUR44SLX2VXQ4MYBE6CYS4OB4RSDMKHA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.383767567 (81h36m6.254850613s) [noise=true , address=145.79.227.33:4870 , piece_id=IBO52LP3RMQQ2U7FYIE6D4LZWOZBTONF2CI3TWVIEUTBT2BGE3EA, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384630530 (81h36m6.255713575s) [noise=true , address=145.79.227.42:4123 , piece_id=BGMQ77X5SJLPQIGM3NS5RGEOIDTECOVR6WKXZQE3XXR7VYTUVL2A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: <nil> Dec 9 01:05:06.384645910 (81h36m6.255728955s) [seg_pos={0 0}, upload=71299]: Segment reached optimal threshold of 13 pieces. Dec 9 01:05:06.384857132 (81h36m6.255940177s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.384864836 (81h36m6.255947881s) [noise=true , address=145.79.227.16:4911 , piece_id=2N46POUCRXNPQCWLHPGHVHAMKLEBPCCMXEQVCKEOYXSJN5EAB62A, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385129441 (81h36m6.256212486s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385137355 (81h36m6.256220401s) [noise=true , address=145.79.227.28:4727 , piece_id=Q7KQPCLR24AEZPUORAZ2DRL4O5JOVD6KLYOVGO64X2MRQ57SC2DQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385319712 (81h36m6.256402768s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 9 01:05:06.385328579 (81h36m6.256411624s) [noise=true , address=45.91.116.5:4298 , piece_id=CZ6QNFBQGUZLIUT56O7LF3VHXZUEVYMXHX74K4YWAJYRJEI4W6SQ, seg_pos={0 0}, upload=71299]: Long tail cancellation due to optimal threshold Dec 9 01:05:06.385349710 (81h36m6.256432755s) [seg_pos={0 0}, upload=71299]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 01:05:06.385438259 (81h36m6.256521304s) [upload=71299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:05:06.385444030 (81h36m6.256527065s) [upload=71299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:06:21.620112808 (81h37m21.491195853s) [upload=71299]: Done uploading segments... Dec 9 01:06:21.620261752 (81h37m21.491344807s) [upload=71299]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.800926144s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X.X...X.X.X....XX...........................................] duration: 3.491144845s trace: https://jaeger.dev.storj.io/trace/24b38e2e3305fb54 FAIL (2025-12-09 01:34:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491168972s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)