Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X................X.........X................] duration: 5.772642288s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:14:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.772688206s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 4.926960701s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:14:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.926992942s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........X..........................X......X................] duration: 3.530493434s trace: https://jaeger.dev.storj.io/trace/5fbc567cfc7ee2e6 FAIL (2025-12-08 02:14:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.530517339s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X................X.........X...............] duration: 7.344603992s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:14:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.344626495s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 7.505895822s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:14:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.505944045s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 7.91094339s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:14:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.911032249s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.014481423s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:14:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.014509947s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.69619825s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 02:14:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.696218659s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X................X.........X..............] duration: 6.027106149s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:14:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.027140695s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 9.05691841s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:13:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.056946492s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.39099907s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:13:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.391031403s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X................X.........X.............] duration: 7.161105267s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:13:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.16113872s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........X..........................X......X...............] duration: 3.426694448s trace: https://jaeger.dev.storj.io/trace/5fbc567cfc7ee2e6 FAIL (2025-12-08 02:13:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42672077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 7.048847898s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:13:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.048874367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.652611663s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:13:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.652641609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.81091924s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 02:13:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.81100887s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X................X.........X............] duration: 6.711464978s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:13:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.711493643s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 4.915854738s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:13:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.915884866s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X..........................X......X..............] duration: 3.463210972s trace: https://jaeger.dev.storj.io/trace/5fbc567cfc7ee2e6 FAIL (2025-12-08 02:12:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463239406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.902965755s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:12:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.902994809s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.049116295s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:12:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.049147896s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X................X.........X...........] duration: 6.008172857s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:12:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.00820547s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.513510254s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:12:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.513540902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.508252662s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:12:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.508747453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.791617796s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 02:12:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.791784512s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X................X.........X..........] duration: 5.770008334s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:12:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.770134965s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X..........................X......X.............] duration: 3.496406722s trace: https://jaeger.dev.storj.io/trace/5fbc567cfc7ee2e6 FAIL (2025-12-08 02:12:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.496439284s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.072628922s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:11:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.072660132s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.869298414s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.869328439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X................X.........X.........] duration: 6.015166324s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:11:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.015190891s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.772732063s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:11:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.772759626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.691119228s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 02:11:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.691149276s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X..........................X......X............] duration: 4.322038612s trace: https://jaeger.dev.storj.io/trace/5fbc567cfc7ee2e6 FAIL (2025-12-08 02:11:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.322106271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.890023314s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:11:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.890059453s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 4.924743474s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:11:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.92477886s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.91003562s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:11:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.910062661s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.204920896s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:11:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.204954929s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.282818406s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 02:11:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.28284629s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.276710661s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:10:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.276737974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X................X.........X........] duration: 7.112446413s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:10:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.112478013s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.680618364s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:10:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.680648241s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X..........................X......X...........] duration: 3.416423031s trace: https://jaeger.dev.storj.io/trace/5fbc567cfc7ee2e6 FAIL (2025-12-08 02:10:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416443511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...................................................X.......] duration: 7.766987154s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:10:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.767021037s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.881186334s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 02:10:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.881218114s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...................................................X......] duration: 7.206956553s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:09:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.20698144s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.201987588s trace: https://jaeger.dev.storj.io/trace/7af096edea7ffcf5 FAIL (2025-12-08 02:09:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.202020801s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X........................X................X.........X.......] duration: 6.387455542s trace: https://jaeger.dev.storj.io/trace/e7164007ac0fd55 FAIL (2025-12-08 02:09:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.387498013s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.677563234s trace: https://jaeger.dev.storj.io/trace/38fc930a38463b73 FAIL (2025-12-08 02:09:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.811888927 (57h59m29.682971962s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812158651 (57h59m29.683241686s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812425138 (57h59m29.683508173s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812681036 (57h59m29.683764071s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.812972881 (57h59m29.684055916s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813282911 (57h59m29.684365946s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813569136 (57h59m29.684652171s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.813933250 (57h59m29.685016285s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814286382 (57h59m29.685369427s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:29.814677757 (57h59m29.685760792s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Uploading piece... Dec 8 01:28:30.638467500 (57h59m30.509550536s) [upload=50956]: Next returned nil segment Dec 8 01:28:30.638500513 (57h59m30.509583548s) [upload=50956]: Waiting for error group managing segments... Dec 8 01:28:31.025088251 (57h59m30.896171286s) [noise=true , address=185.167.204.205:4007 , piece_id=664L7FGKSK5ACF5TNYFRLLXT6PKB5WD77TYNDRJTRX4GPBKW64EA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.031525057 (57h59m30.902608092s) [noise=true , address=185.167.204.205:4022 , piece_id=G7YU6BM6JTSQCTPUQVSDYNBF7AERYHIMPV6RJC7UMPWS7KHFB6QQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.032013847 (57h59m30.903096882s) [noise=true , address=185.167.204.206:4040 , piece_id=QUJN7KGAWEV3EMPJIQEB246EOYFCJOERPIXX6E7TV3XT3BN3WFLA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033542528 (57h59m30.904625573s) [noise=true , address=185.167.204.206:4067 , piece_id=RWXUAI7ZSKVDSNCGZVWZNSER2KDNEJWSXICOTI5LJJNK3VBQUNIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.033594277 (57h59m30.904677322s) [noise=true , address=185.167.204.206:4043 , piece_id=E45RUPDKI24Q7W47TJFPEIZJRPWLUB7NZRJOWX6WAXXCENYJQG6Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.035521096 (57h59m30.906604131s) [noise=true , address=185.167.204.205:4003 , piece_id=PQKECUXLAZSRP35HV5B2CQ7LKVIRSUKIJGQHN23BH6BUATZQDB3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.041794511 (57h59m30.912877547s) [noise=true , address=185.167.204.206:4044 , piece_id=TGEJCREVPMY3IR2UEH7RSGVLB2WTWR6E764FDBMLPHS4AMIZC7WA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.042923371 (57h59m30.914006406s) [noise=true , address=185.167.204.206:4056 , piece_id=RZANF6DB5CC6BT7CG5FEIR556DIJRNFWTLX7FT55NMYNEJZGCANA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043190219 (57h59m30.914273264s) [noise=true , address=185.167.204.206:4055 , piece_id=UTFF6JURJV66ELHF437DNZFGCC3OFC2PPFH4WXIJHR35WBPIG2ZA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.043692316 (57h59m30.914775361s) [noise=true , address=185.167.204.205:4026 , piece_id=36OD7XBYHINMQGJGODBYDHSCNRI6W4KIM53OEWV2K3YXWHSUN5IA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.045169658 (57h59m30.916252693s) [noise=true , address=185.167.204.205:4002 , piece_id=T47YMGI6QYX6AIG6ES6W4UQUADHW35MR4EXNLK3KLL3KTGUT4L3Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052531105 (57h59m30.92361415s) [noise=true , address=185.167.204.205:4032 , piece_id=CSSYNAHURAHYAQATYYUESWH4OMCFKOB5ZRDRXMR4LKSHJTB3QHBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.052934233 (57h59m30.924017278s) [noise=true , address=185.167.204.205:4012 , piece_id=LVVLB4WXA2DS3YMCKW666XZJOIHRGD4TY6FMUQINMQE5GIOCQBIA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.533128384 (57h59m31.404211419s) [noise=true , address=81.171.14.70:4039 , piece_id=N6JKYD5H2H75ML62X6AQUZUPHESFOTQYX343P6JGEHLSELFOGXBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.558660150 (57h59m31.429743185s) [noise=true , address=81.171.14.70:4026 , piece_id=OGMWFHMHMZZVULRWPWUJMG3JOZJSCCFNDUVCISRWENXDM3V7AIGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.575319403 (57h59m31.446402438s) [noise=true , address=81.171.14.70:4044 , piece_id=WPT6HRQ5CK7LR2MS5A4BTX57UC2KT7KEYWMREEE6FC2VKIWANTGQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.604666905 (57h59m31.47574995s) [noise=true , address=81.171.14.70:4045 , piece_id=O4QVYEC6WVPT4W7CYFBKINLTFXBZWWDDTKCGJXBYFCOBB4XOMNJQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.606191879 (57h59m31.477274924s) [noise=true , address=81.171.14.70:4033 , piece_id=64GND7UPVOQTQRD2OQNMBRXLIM4DKMJQ3C64UECT6II54EZJFH2Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607295470 (57h59m31.478378505s) [noise=true , address=81.171.14.70:4046 , piece_id=VRSVA6MDMFP35V6CQIYOYI6WOWOB4AFOGSQWH5KZLRPBZEYV52TA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.607829497 (57h59m31.478912532s) [noise=true , address=81.171.14.70:4043 , piece_id=P25PGASSBJNQAMVBESXNZ7B2BL2KV4LVRXMNX5AMXBTJEBSLC54Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.622814972 (57h59m31.493898017s) [noise=true , address=81.171.14.70:4041 , piece_id=Y3CZ3KJBFUHF4NUFGOOAADVZLWZGCZU3SJYPDJPA3OXSGALIHH5Q, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:31.700677745 (57h59m31.57176078s) [noise=true , address=81.171.14.70:4031 , piece_id=K4CVKEX76ZZYOAA3M7P3NREY3FUTDJ4WXDLTGNHCEWLIUNRQFS4A, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.138330650 (57h59m32.009413685s) [noise=true , address=89.149.201.147:4019 , piece_id=522WNYYPXCCTKWH7IJ3AJ7VPOLY3TJG2LX4D2YPQ5H7JMMZRWUGA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.278378073 (57h59m32.149461118s) [noise=true , address=89.149.201.147:4008 , piece_id=VSXPGW4NK6CWZI55NDZXYZ6W6OXME65F74XKFJRLOBJHJUS4WMBQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.286774140 (57h59m32.157857185s) [noise=true , address=89.149.201.147:4009 , piece_id=NEDO736SNAEI6YW33G7ADUVHOJ3NEN64JPF2PSF4457YG643MPEQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288000365 (57h59m32.15908341s) [noise=true , address=89.149.201.147:4012 , piece_id=LBWBQX5DKFTPZJNVB4LBKB5U5F7WOEUTENYAZNP6RP4TOYOMDNYA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: <nil> Dec 8 01:28:32.288021095 (57h59m32.15910413s) [seg_pos={0 0}, upload=50956]: Segment reached optimal threshold of 26 pieces. Dec 8 01:28:32.288187731 (57h59m32.159270767s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288194285 (57h59m32.15927732s) [noise=true , address=89.149.201.147:4024 , piece_id=LMOQLMMO7YVCXFCMSU2UTYHIUS2MTLIADVJ4JUHIOBM4KZYNBCHA, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288457505 (57h59m32.15954054s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u3qrydva66a46yxwwL8KhP22RKxyWH321urJzP6wn7tCiDmhr): protocol: expected piece hash; context canceled; EOF Dec 8 01:28:32.288467023 (57h59m32.159550058s) [noise=true , address=89.149.201.147:4004 , piece_id=DERCLIL5DZ2KTOJPPAIMAAW5YTG2XKESQF2FUIC3NKR7NNYUQZQQ, seg_pos={0 0}, upload=50956]: Long tail cancellation due to optimal threshold Dec 8 01:28:32.288488875 (57h59m32.15957192s) [seg_pos={0 0}, upload=50956]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 01:28:32.288593625 (57h59m32.15967667s) [upload=50956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:28:32.288598955 (57h59m32.15968199s) [upload=50956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:32:18.393065902 (58h3m18.264148947s) [upload=50956]: Done uploading segments... Dec 8 01:32:18.393181121 (58h3m18.264264166s) [upload=50956]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.677592329s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.984568907s trace: https://jaeger.dev.storj.io/trace/57258282a8e974fd FAIL (2025-12-08 02:09:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.202906305 (58h26m7.07398934s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203371201 (58h26m7.074454246s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.203861354 (58h26m7.07494439s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204290401 (58h26m7.075373446s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.204711363 (58h26m7.075794398s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205118208 (58h26m7.076201253s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.205554277 (58h26m7.076637323s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206116137 (58h26m7.077199182s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.206664562 (58h26m7.077747607s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207102556 (58h26m7.078185601s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207519911 (58h26m7.078602956s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.207967393 (58h26m7.079050438s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208416427 (58h26m7.079499472s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.208830746 (58h26m7.079913791s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209255245 (58h26m7.080338291s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.209662921 (58h26m7.080745966s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:07.214493680 (58h26m7.085576725s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Uploading piece... Dec 8 01:55:08.555121654 (58h26m8.426204699s) [upload=51329]: Next returned nil segment Dec 8 01:55:08.555145519 (58h26m8.426228564s) [upload=51329]: Waiting for error group managing segments... Dec 8 01:55:08.935193756 (58h26m8.806276791s) [noise=true , address=64.49.13.27:4957 , piece_id=7METNUA4TAJ2FSYO5S4NXECVMMMCXKFL5CAHPQBQJ6WREQ3YFC3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.093051314 (58h26m8.964134359s) [noise=true , address=145.79.227.16:4933 , piece_id=LHF64L3SWXFVZ5X5D67I44WSJNKRFB5G4YP7DFHDWOAYE23II3OA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.098812273 (58h26m8.969895318s) [noise=true , address=145.79.227.18:4010 , piece_id=S2EOTEV4TBNNDOBN7TIX76AJE2WI5P5DM3JK2NJB25TISYLS7VSQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.102616718 (58h26m8.973699763s) [noise=true , address=145.79.227.25:4627 , piece_id=K7UYU3W2CACUAQUUL72GG5F7DP5CHAHZ4M47TZG6NBN5Z6ZCDRBA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.118902949 (58h26m8.989985994s) [noise=true , address=145.79.227.34:4928 , piece_id=DRKF2DRSCBAYSZO66A2ZON2JS6HIJIT2YUWFZRS3W2CKFTV3NLVA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.124783838 (58h26m8.995866883s) [noise=true , address=77.90.31.18:4825 , piece_id=EJ4LO4TCEFB4OQQLFOEUYGQKK462ZPXXEIIY5CHF2PERXWDGQNDA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.126718282 (58h26m8.997801327s) [noise=true , address=145.79.227.11:4740 , piece_id=KL54LL2UQRT6MH2K3NQQJB3BKPX4QRWQMXRK27MW2U7CHFDCCWFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.145158485 (58h26m9.01624153s) [noise=true , address=77.90.31.10:4550 , piece_id=PDCZ7VXNHQJ6LYPG64GZRYP4MRBGVPXCWTXSRT3UV2N6HGMDE2KA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.156650047 (58h26m9.027733092s) [noise=true , address=145.79.227.15:4908 , piece_id=JIUZSMOAKBFCFAX6RQ52TCNZMYBREQCYOEM75U3H2QYR4WH3KWCA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.162020363 (58h26m9.033103398s) [noise=true , address=77.90.31.10:4539 , piece_id=27YFK55QLZDEULYABQ3OBDSPDDEIUJ2MJSD7X3A4K6HRFJMRQJUA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.198899907 (58h26m9.069982942s) [noise=true , address=77.90.31.12:4621 , piece_id=S4FVJHEPIO2MVO56ZRIJ27RMTV723MERPLJLVD2T2GAO3T7K24TA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.208468337 (58h26m9.079551382s) [noise=true , address=45.91.116.4:4945 , piece_id=6UNWQQAWWWWC2XVUCDRCTAYJDTHBXAZJMXA6KLCTOP5PYSAJCCGA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.212217697 (58h26m9.083300732s) [noise=true , address=45.91.116.12:4254 , piece_id=UCKU4TBXJC4PSYHRDPUXETXLAILDOWA5NN7RCVVNYLVUBY7FBIFA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215832390 (58h26m9.086915435s) [noise=true , address=77.90.31.11:4579 , piece_id=JINRNEVUAP2VI5Y4CR7ALWDQIAIAFX3SP2VVZSXDDG6XPUKASE3Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: <nil> Dec 8 01:55:09.215862086 (58h26m9.086945161s) [seg_pos={0 0}, upload=51329]: Segment reached optimal threshold of 14 pieces. Dec 8 01:55:09.216078098 (58h26m9.087161133s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216085662 (58h26m9.087168707s) [noise=true , address=45.91.116.6:4003 , piece_id=CRONTV2B2AVZTLKU3YNIEGQZSON7XHHL7S3K6KNNJV4YR5FHQ3PA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216307705 (58h26m9.08739075s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216313956 (58h26m9.087396991s) [noise=true , address=145.79.227.30:4786 , piece_id=VM65RUN3GUZZFUOHFYE7ZLPWU6COCQUMY3IK7GCNWPIUZAMEBR7Q, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216414809 (58h26m9.087497844s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216420681 (58h26m9.087503726s) [noise=true , address=77.90.31.16:4760 , piece_id=Q67LYTDJCAR2TFVSAIIYCUYM7D44JFPBLCSKRNLXH26EAW5UQ3GA, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216562861 (58h26m9.087645906s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Dec 8 01:55:09.216570025 (58h26m9.08765307s) [noise=true , address=45.91.116.7:4037 , piece_id=UBNFDSXQBISC5QQSNGAAZIR7LRE7FCYRVYN5MMOY3A2ALEOQBYQQ, seg_pos={0 0}, upload=51329]: Long tail cancellation due to optimal threshold Dec 8 01:55:09.216585564 (58h26m9.087668609s) [seg_pos={0 0}, upload=51329]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 01:55:09.216654365 (58h26m9.0877374s) [upload=51329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 01:55:09.216659064 (58h26m9.087742099s) [upload=51329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 01:55:35.852213564 (58h26m35.723296629s) [upload=51329]: Done uploading segments... Dec 8 01:55:35.852273619 (58h26m35.723356664s) [upload=51329]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.984629483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...................................................X.....] duration: 6.964294272s trace: https://jaeger.dev.storj.io/trace/27d92c196f473eec FAIL (2025-12-08 02:09:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4891s) [upload=51469]: Got next segment. Inline: false Dec 8 02:04:21.143134547 (58h35m21.014217592s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 02:04:21.143141259 (58h35m21.014224304s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 02:04:22.267502706 (58h35m22.138585751s) [seg_pos={0 0}, upload=51469]: Begin upload segment... Dec 8 02:04:22.267624127 (58h35m22.138707172s) [seg_pos={0 0}, upload=51469]: Done begin upload segment. Dec 8 02:04:22.267698027 (58h35m22.138781073s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.267712997 (58h35m22.138796042s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268434680 (58h35m22.139517715s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268550331 (58h35m22.139633366s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268913111 (58h35m22.139996146s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.268924032 (58h35m22.140007067s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269398886 (58h35m22.140481921s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269414686 (58h35m22.140497721s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.269753261 (58h35m22.140836296s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270016322 (58h35m22.141099367s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270259175 (58h35m22.14134222s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.270655339 (58h35m22.141738374s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271143158 (58h35m22.142246412s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271230565 (58h35m22.14231361s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.271662086 (58h35m22.142745121s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:22.272592658 (58h35m22.143675703s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Uploading piece... Dec 8 02:04:24.009199123 (58h35m23.880282158s) [upload=51469]: Next returned nil segment Dec 8 02:04:24.009221235 (58h35m23.88030428s) [upload=51469]: Waiting for error group managing segments... Dec 8 02:04:24.293906896 (58h35m24.164989931s) [noise=true , address=145.79.227.24:4583 , piece_id=WYTXGXG77ZW4RGWX7YWDU2NH6R5CCPVBPQH3ISHUAJVKZJRONS5Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.298576738 (58h35m24.169659783s) [noise=true , address=145.79.227.28:4727 , piece_id=UYRK457JBF2AHCIXZ4XOIL7YHOVYZ7IPDUTB6WOZZUDEW5Q7NNCQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.301356571 (58h35m24.172439616s) [noise=true , address=145.79.227.25:4619 , piece_id=XBNGVRIQLQUWKKBF4ULL7ZJDDT75QLY7ES2J7WHYWRD2ZLHOGU2Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.326466765 (58h35m24.19754981s) [noise=true , address=145.79.227.16:4910 , piece_id=ETE7ZBHT5IJUOBWCA5BGLKC2ANDEUB2M3DOMXWWS2RBWSJQBARRQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.328633530 (58h35m24.199716586s) [noise=true , address=145.79.227.26:4653 , piece_id=DKW72PEWPIOHPYHLI54PBU7NR3HGUBLFM3MN4XQQFWIIZ4FHIUNA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.360508106 (58h35m24.231591151s) [noise=true , address=145.79.227.40:4051 , piece_id=7CQTGHFZ542WUHYX2CTOPN4PF2KDOOLDGV64THDBZCYOCXA3JWDA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361116263 (58h35m24.232199308s) [noise=true , address=145.79.227.25:4617 , piece_id=S4ETCJ6WMCPXXQCYPPBBMHLMK2SVJV4S5VOKHGTDAGSGG6WRYH3Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.361695086 (58h35m24.232778131s) [noise=true , address=145.79.227.41:4087 , piece_id=XO6K3CXK7KZ5VEVXULHKQXLSSLIIWIX3VKQOSD7HY25WMNMCDLEQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.371520525 (58h35m24.24260356s) [noise=true , address=145.79.227.38:4977 , piece_id=73WXA33TWM4MFVTHDHW3PXJVABMCVMQYI3P42AHT3OYPKMXIVTUQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.381271893 (58h35m24.252354928s) [noise=true , address=145.79.227.23:4545 , piece_id=NBYCSU2KSHTAL223AHKHREQWKXUEQKJU46TY3SPYTKT5N5T2OUSA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386403053 (58h35m24.257486098s) [noise=true , address=77.90.31.11:4554 , piece_id=BQP7XTRA2SBVINT4J7JAOF3ORNENBBIQN4ABMCEMQQ6FQL6QUOKA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.386427209 (58h35m24.257510254s) [noise=true , address=77.90.31.10:4518 , piece_id=6AR3NAFODP54SKG3SJSJSIQQFOYXO2C65WVNN4Z4XG426ZEOEDQA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400489386 (58h35m24.271572431s) [noise=true , address=77.90.31.13:4628 , piece_id=LWTOCQRRA7HLI5ZPPRQ25IM22LMQNRR5H3TTOTSTGCFOYFZMABAA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: <nil> Dec 8 02:04:24.400513903 (58h35m24.271596938s) [seg_pos={0 0}, upload=51469]: Segment reached optimal threshold of 13 pieces. Dec 8 02:04:24.400813233 (58h35m24.271896288s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39434->145.79.227.39:4013: use of closed network connection Dec 8 02:04:24.400825677 (58h35m24.271908722s) [noise=true , address=145.79.227.39:4013 , piece_id=UI22RDXJ2DY3RFNSJVNT43XQJ6B2S3EO6SFLYHXW4VPVQJSYYF6Q, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.400873027 (58h35m24.271956082s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47926->145.79.227.22:4511: use of closed network connection Dec 8 02:04:24.400894248 (58h35m24.271977293s) [noise=true , address=145.79.227.22:4511 , piece_id=GGA463DAOSZCW7SQGZW2LGFRQVNUOUJ2L2GXRXLSDPC65KVPY3TA, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401247129 (58h35m24.272330184s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: read tcp 188.166.225.170:43344->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 02:04:24.401260003 (58h35m24.272343059s) [noise=true , address=145.79.227.23:4547 , piece_id=5Y6V5QWLCWNOLT2GDNUEIRSG75KGNGIFLBCBGSIT7XH46RZYVDGQ, seg_pos={0 0}, upload=51469]: Long tail cancellation due to optimal threshold Dec 8 02:04:24.401285452 (58h35m24.272368497s) [seg_pos={0 0}, upload=51469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 02:04:24.401382247 (58h35m24.272465292s) [upload=51469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 02:04:24.401391394 (58h35m24.272474429s) [upload=51469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 02:05:39.049922629 (58h36m38.921005704s) [upload=51469]: Done uploading segments... Dec 8 02:05:39.050003342 (58h36m38.921086377s) [upload=51469]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.964353385s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)